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Purpose of the Study and Intended Audience 

When Stanford's Artificial Intelligence MOOC made headline news in 2011, one of the early predictions 
was that quality education at mass scale and at low cost was around the corner. Given our research 
center's interest in the productivity of educational interventions, we have been watching for evidence 
that MOOCs are cost-effective in producing desirable educational outcomes compared to face-to-face 
experiences or other online interventions. While the MOOC phenomenon is not mature enough to 
afford conclusions on the question of long-term cost-effectiveness, this study serves as an exploration of 
the goals of institutions creating or adopting MOOCs and how these institutions define effectiveness of 
their MOOC initiatives. We assess the current evidence regarding whether and how these goals are 
being achieved and at what cost, and we review expectations regarding the role of MOOCs in education 
over the next five years. 

We expect this report to be of interest to administrators, faculty members, and other instructors at 
colleges, universities, and other educating institutions who are wondering whether MOOCs can advance 
their missions and serve learners well; to researchers in the fields of educational technology and higher 
education who are contemplating what kinds of research questions might be useful to address; and to 
policymakers, investors, and funders considering whether MOOCs are a worthwhile investment of often 
limited resources for education. 
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Executive Summary 


Over the past few years, observers of higher education have speculated about dramatic changes that 
must occur to accommodate more learners at lower costs and to facilitate a shift away from the 
accumulation of knowledge to the acquisition of a variety of cognitive and non-cognitive skills. All 
scenarios feature a major role for technology and online learning. Massive open online courses (MOOCs) 
are the most recent candidates being pushed forward to fulfill these ambitious goals. To date, there has 
been little evidence collected that would allow an assessment of whether MOOCs do indeed provide a 
cost-effective mechanism for producing desirable educational outcomes at scale. It is not even clear that 
these are the goals of those institutions offering MOOCs. This report investigates the actual goals of 
institutions creating MOOCs or integrating them into their programs, and reviews the current evidence 
regarding whether and how these goals are being achieved, and at what cost. 

How and Why are Institutions Engaging with MOOCs? 

Through interviews with 83 administrators, faculty members, researchers, and other actors from 62 
different institutions (see Appendices I, III and VI for details) active in the MOOCspace or more generally 
in online learning, we observed that colleges and universities have adopted several different stances 
towards engaging with MOOCs and are using them as vehicles to pursue multiple goals. Some 
institutions are actively developing MOOCs and may be termed "producers," some are using MOOCs 
developed by other institutions in their programs and could be termed "consumers," and a few are 
doing both. Others are adopting a "wait-and-see" approach, or have considered MOOCs and have 
decided against any form of official engagement. There is no doubt, however, that the advent of MOOCs 
has precipitated many institutions to consider or revisit their strategy with respect to online learning, 
whether at large scale or small. 

Among the 29 institutions in our sample that were already offering or using MOOCs in some way, we 
identified six major goals for MOOC initiatives: 

• Extending the reach of the institution and access to education 

• Building and maintaining brand 

• Improving economics by lowering costs or increasing revenues 

• Improving educational outcomes for both MOOC participants and on-campus students 

• Innovation in teaching and learning 

• Conducting research on teaching and learning. 

The frequency with which each goal was mentioned by the 29 institutions offering or using MOOCs is 
shown in table ESI. 

Goal 1: Extending Reach and Access 

Extending the reach of the institution to a wider audience and improving access to education was the 
most commonly identified goal for offering a MOOC, mentioned by 65% of the institutions in our study 
that were offering or using MOOCs and 42% of our interviewees overall. Ways in which MOOCs are 
expected to increase access to education include: 

• "Broadcasting" to global audiences (see Case 1 ) 

• Alleviating infrastructure constraints domestically and in rapidly developing countries where the 
existing physical plant cannot accommodate the growing demand for postsecondary education 
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• Easing the pressure on oversubscribed programs or "bottleneck" courses 

• Increasing access to instructors skilled in specialized domains and niche subjects 

• Providing flexibility in time and place of study 

• Providing a no-risk, low-cost option for at-risk students in developmental education, setting 
them on an accelerated path to credit-bearing courses and more timely completion of a degree 
(see Case 2 ) 

• Flexibility for students to create their own programs with courses from various institutions 

• Continuing education or "professional development" for alumni and other working adults. 

Data from MOOC platforms indicate that MOOCs are providing educational opportunities to millions of 
individuals across the world. However, most MOOC participants are already well-educated and 
employed, and only a small fraction of them fully engages with the courses. Overall, the evidence 
suggests that MOOCs are currently falling far short of "democratizing" education and may, for now, be 
doing more to increase gaps in access to education than to diminish them. 

Table ES 1: Institutional Goals for Developing and Delivering or Using MOOCs 


Institutional Goal 

% of institutions offering /using 
MOOCs stating this as a goal 
(n=29)* 

% of all interviewees who 
raised this as a goal 
(n=83) 

Extending Reach and Access 

65% 

42% 

Building and Maintaining Brand 

41% 

25% 

Improving Economics 

38% 

29% 

Improving Educational Outcomes 

38% 

20% 

Innovation 

38% 

19% 

Research on Teaching and Learning 

28% 

18% 


* Includes one museum in addition to the universities and colleges. 


Goal 2: Building and Maintaining Brand 

For institutions of higher education, building and maintaining brand serves to attract and retain 
students, faculty members, and partnership opportunities with other institutions, funders, and alumni 
networks. Interviewees from 41% of the institutions in our study sample that were offering or using 
MOOCs stated that branding, positioning, or attracting students was a strategic goal for the initiative. 
None of the community colleges mentioned branding as a goal for engaging in MOOCs. 

Institutions that were employing MOOCs as a vehicle to expand their brand made strategic decisions to 
showcase programs, specialties, and research capabilities in which they believed they were leaders or 
wanted to be among the first to stake a claim to world-class expertise. Case 3 and Case 4 provide 
examples of how institutions are using MOOCs to increase visibility and recruitment. While many 
institutions have received significant media attention as a result of their MOOC activities, isolating and 
measuring impact of any new initiative on brand is a difficult exercise. Most institutions are only just 
beginning to think about how to capture and quantify branding-related benefits, for example, by 
comparing historical data on applications and admissions with post-MOOC statistics. Selective 
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institutions must balance the apparently contradictory goals of increasing access to their offerings with 
building and maintaining brand. 

Goal 3: Improving Economics: Reducing Costs or Increasing Revenues 

Thirty-eight percent of the institutions in our sample that were offering or using MOOCs and twenty- 
nine percent of our interviewees overall claimed that a goal for MOOC initiatives was to lower costs or 
increase revenues, or both. These objectives appeared to be more important to public universities than 
to private ones. There was widespread acknowledgement that the current expenditures on MOOC 
development could not continue indefinitely without financial justification. 

Potential Cost Savings from MOOCs 

In spite of recognition that MOOCs have, so far, proved to be a significant drain on time and money for 
institutions, interviewees offered several possibilities for eventual cost savings: 

• Re-using MOOC materials multiple times 

• Sharing MOOC materials across instructors and campuses 

• Developing common courses to offer across institutions 

• Replacing on-campus courses with MOOCs 

• Faculty time savings 

• Reducing the need for facilities 

• Recruitment efficiencies 

• Less costly student support services provided by non-faculty members (see Case 5 ) 

• Increasing student throughput. 

MOOCs as a Source of Revenue 

While a small number of interviewees (less than 5%) speculated that some institutions of higher 
education are pursuing MOOCs as a potential source of revenue, only two representatives from state 
universities explicitly stated revenue generation as a goal. Most institutions, at least for now, appear to 
be treating MOOCs as an investment. Potential current and future sources of revenue included: 

• Offering credit and charging tuition 

• Creating new for-fee courses and programs 

• Drawing MOOC participants into existing, full-tuition degree programs 

• Increasing class sizes 

• Licensing fees for use of MOOC materials or data by other institutions 

• Fees for additional services, for example, online tutoring, face-to-face instruction with a local 
instructor 

• Grant revenues 

• Matchmaking or training for employers 

It appears that revenue streams for MOOCs are slowly materializing but we do not expect the costs of 
MOOC production to fall significantly given the highly labor-intensive nature of the process. While these 
costs may be amortized across multiple uses and multiple years, they will still be additive costs to the 
institutions creating MOOCs. Free, non-credit bearing MOOCs are likely to remain available only from 
the wealthiest institutions that can subsidize the costs from other sources of funds. For most 
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institutions, ongoing participation in the current MOOC experimentation will be unaffordable unless 
they can offer credentials of economic value to attract fee-paying participants, or can use MOOCs to 
replace traditional offerings more efficiently, most likely by reducing expensive personnel. 

Goal 4: Improving Educational Outcomes 

Thirty-eight percent of the institutions participating in our study that were offering or using MOOCs, and 
twenty percent of our interviewees overall, expected MOOCs to lead to an improvement in educational 
outcomes, some directly within the MOOC format and some indirectly through the transfer of new 
strategies and techniques to on-campus teaching. Strategies for improving educational outcomes 
included: 

• Motivating instructors to rethink pedagogy 

• Course re-design 

• "Chunking" lectures and interspersing questions 

• Fine-tuning instructional materials 

• Providing instant feedback to course participants 

• Gamification and badging to increase motivation 

• Outreach to course participants to encourage persistence 

• Adaptive learning/personalization/mastery-based learning 

• Using MOOCs in K-12 to prepare students for college. 

While interviewees provided many examples of how MOOCs have been used to change instruction, for 
the most part, actual impact on educational outcomes has not been documented in any rigorous 
fashion. Consequently, in most cases, it is unclear whether the goal of improving educational outcomes 
has been achieved. Case 6 and Case 7 highlight two exceptions where instructional changes and their 
effects on student performance have been carefully recorded, providing evidence of improvement in 
student performance as a result of adopting MOOC strategies in on-campus courses, such as frequent 
assessment and automatic feedback, or of integrating MOOCs into flipped on-campus courses. 

Goal 5: Innovation in Teaching and Learning 

Thirty-eight percent of the institutions in our study that were engaging with MOOCs, and nineteen 
percent of our interviewees overall, presented MOOCs as vehicles for experimenting and innovating 
with pedagogy (e.g., Case 8) and new models of higher education. While these institutions specified 
innovation as an end goal in itself, we note that the other five goals we report involve innovating as a 
means to other ends. A number of interviewees portrayed experimentation with MOOCs and online 
learning as preparation for an uncertain future. 

The large number of courses that have been developed or substantively re-designed since the 
appearance of MOOCs stands as evidence of innovation, for example, MOOC. list currently catalogs 
around 1,700 courses. It is abundantly clear that MOOCs have prompted many institutions and faculty 
members to engage in new educational activities. The strategies employed online such as frequent 
assessments and short lectures interspersed with questions are being taken back on-campus. It is less 
clear what has been gained by these new initiatives because the value of innovation is hard to measure 
unless it can be tied to a further, more tangible objective. We address the evidence of success in using 
MOOCs to achieve other goals in relevant sections of this report but conclude that most institutions are 
not yet making any rigorous attempt to assess whether MOOCs are more or less effective than other 
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strategies to achieve these goals. A few are starting to develop metrics to assess the impact of MOOC- 
related innovations on various objectives. 

Goal 6: Research on Teaching and Learning 

Research on teaching and learning was stated as a goal for MOOC initiatives by 28% of the 29 
institutions offering or using MOOCs and by 18% of our 83 interviewees. Areas in which interviewees 
described research explorations and advances included: 

• The role of social media networks in teaching and learning 

• Testing pedagogical strategies 

• Student engagement and motivation 

• Machine learning/modeling research 

• Natural Language processing 

• Human-computer interaction 

• Personalized/adaptive learning 

• Comparing hybrid with traditional courses 

• Developing data standards and a common platform for data mining. 

A great deal of effort is being expended on trying to improve participant engagement and completion of 
MOOCs and less effort on determining whether participants actually gain skills or knowledge from the 
courses. Case 9 describes the work of one of the few research groups focusing on the latter. It was, 
however, noted that this question of efficacy is rarely asked with respect to regular college courses. 

Most instructors and researchers appear content with equating the completion of a MOOC with having 
learned something. It was apparent that progress in using MOOCs to improve teaching and learning is 
being impeded by difficulty using the platform data and lack of clarity regarding regulations applicable 
to the participants and their data. While the potential for MOOCs to contribute significantly to the 
development of personalized and adaptive learning is high, the reality is far from being achieved. A great 
deal of coordination and collaboration among content experts, instructors, researchers, instructional 
designers, and programmers will be necessary to result in meaningful improvements to teaching and 
learning. 

Resource Requirements and Costs of MOOCs 

While the evidence is mixed as to whether MOOCs have helped institutions achieve their stated goals, it 
is important to monitor the costs of developing and delivering MOOCs to allow a comparison with 
alternative means to achieving these goals. While interviewees provided rough estimates of such costs 
ranging from as low as $5,000 per MOOC to as high as $1.2 million, we conducted cost analyses of 
MOOC production and delivery at four different institutions to obtain reliable estimates. 

The major cost drivers in MOOC production and delivery are: 

• Number of faculty members, administrators, and instructional support personnel involved: 
MOOC production teams seldom included fewer than five professionals and, in at least one 
instance described to us, over 30 people were involved. Faculty members typically reported 
spending several hundred hours in the production and delivery of a single MOOC. 

• Quality of videography; we estimated costs for high quality video production at $4,300 per hour 
of finished video, using national average prices. 
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• The nature of the delivery platform 

• Technical support for participants 

• Programming for special features such as computer code auto-graders, virtual labs, simulations, 
or gamification 

• Analysis of platform data 

Using the ingredients method (Levin & McEwan, 2001), we estimated the actual costs of MOOC 
production and delivery at three universities and one museum (see Case 10, Case 11. Case 12, and Case 
13 ). Personnel costs alone ranged from $29,000 to $244,000 per MOOC (using national average salaries 
and benefits), depending on the number of people involved in the production process, the quality of 
video production, and the amount of time dedicated. Assuming personnel costs account for 75% of total 
costs once facilities, equipment, and overhead are included, we estimate total costs per MOOC of 
$39,000 - $325,300, as shown in Table ES2. Costs per completer in the two cases where completion data 
were available were $74 and $272. By comparison, Bates (personal communication, April 29, 2014; 

Bates & Sangra, 2011, pp. 163-166) currently estimates the costs to develop a regular 3-credit online 
course delivered via a learning management system at $35,000 - $50,000. 

Table ES 2: Estimated Costs of MOOC Production and Delivery at Four Institutions 


Institution 

Type of MOOC 

Length of MOOC 
(weeks) 

Total estimated costs 
per MOOC 

Costs per 
completer 

Teachers College, 
Columbia University 

xMOOC 

8 

$38,980 

$74 

University of Manitoba 

cMOOC 

12 

$65,800 -$71,800 

* 

American Museum of 
Natural History 

xMOOC 

4 

$104,620 

$272 

Large Midwestern 
University 

xMOOC 

5-8 

$203,770 -$325,330 

* 


* Completion data were not available for these MOOCs. See Cases 10, 11, 12, and 13 for sources. 


The question for MOOCs over the longer term is whether variable costs can be contained by automating 
functions and substituting instructional support provided by expensive faculty members with less costly 
teaching assistants, part-time instructors, or peer-to-peer learning and assessment. Many MOOC 
developers anticipate that the costs of re-running a MOOC will be substantially lower than the costs of 
initial development. In Case 10, we show that the costs of re-running a cMOOC were 38% lower than the 
cost of the initial offering. However, given the intense level of instructor involvement in cMOOCs, this is 
unlikely to be a useful predictor for xMOOC re-runs where instructor involvement may be minimal or 
absent. A similar cost analysis for xMOOC re-runs is much needed. 

The Future of Higher Education and the Potential Role of MOOCs 

Many interviewees envisaged a future of more "unbundling" of educational services, more choice for 
students in how their education is delivered, greater price competition among providers, and greater 
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use of technology in education. Several questioned whether two- or four-year degrees are the most 
useful way to acquire an education, and suggested that while colleges and universities may continue to 
provide foundational skills and knowledge, the flexibility of online education will allow for more "just-in- 
time" learning experiences throughout an individual's career. 

Most interviewees believed that higher education will be increasingly pressured to shift towards 
competency-based models of credentialing. Several foresaw the possibility that the emphasis on earning 
a degree will dwindle in favor of an emphasis on demonstrating capabilities needed by employers. The 
possibility was raised that federal funding for postsecondary education could shift away from paying for 
access to college-based degrees, and towards paying for the demonstration of competency through 
individually accredited educational experiences. Specific ideas about the potential role of MOOCs in the 
future of higher education are summarized in Table ES3. 

Conclusions 

While much of the media hype surrounding MOOCs may be subsiding and only a small percentage of 
institutions of higher education are actually offering MOOCs (5% according to Allen & Seaman, 2014), it 
is clear that the infrastructure and effort that has been poured into such initiatives are not likely to 
evaporate overnight. Whether MOOCs as they currently stand persist into the future is certainly 
debatable, but there is no doubt that online and hybrid learning is here to stay and that MOOCs have 
catalyzed a shift in stance by some of the most strongly branded institutions in the United States and 
abroad. 

We ourselves expect that MOOCs or their derivatives will continue to play a role in the continuing 
education of working professionals, in experimentation with various types of blended or hybrid delivery 
models on-campus, and in efforts to help struggling students find low-risk options to build skills that 
allow them to test out of developmental education courses. MOOCs could potentially affect higher 
education in more revolutionary ways, but not without some significant changes to the status quo. For 
example, if MOOC providers are able to offer participants credentials of economic value (e.g., college or 
high school credits; verified certificates of accomplishment; virtual badges to certify skills or non- 
cognitive traits), a market will be established for individual courses, which could be extended to a 
variety of non-degree-based educational experiences. Such a market would greatly benefit from a 
system for evaluating and accrediting each course or educational experience - one that is trusted by 
employers, educators, and funding agencies alike, and that reflects a learner's ability to contribute 
productively to society. If funding agencies subsequently become willing to allow learners to apply 
financial aid to any such recognized educational experience, the landscape of higher education will be 
opened to more competition, leading to lower costs. 

A second way in which MOOCs may contribute to revolutionary change is in their potential to catalyze 
the development of true adaptive learning experiences. However, this will require a significant 
investment of resources, cross-institutional and inter-disciplinary collaborations, and clarification or 
change in regulations applying to data use and sharing. If the development of course markets is a trend 
that may threaten quality of the overall educational experience but lower costs, the development of 
adaptive learning could balance the shift by improving the quality of educational experiences delivered 
at scale. 
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Table ES 3: Ideas Raised by Interviewees about MOOCs in the Future of Education 


Ideas Raised by Interviewees about MOOCs in the Future of Education 

Mentions 
among 83 
interviewees 

How MOOCs will evolve as courses 

MOOCs will serve as educational resources rather than as stand-alone courses 

Many 

MOOCs will be targeted at specific audiences 

Many 

MOOCs will offer revenue-generating services such as tutoring, face-to-face 
interactions, and study groups 

Many 

MOOCs will offer credentials of economic value such as high school or college credits, 
or badges that are recognized by employers 

Some 

Learning analytics data will catalyze the advent of personalized, adaptive, and mastery- 
based learning 

A few 

Convergence of cMOOC and xMOOC pedagogy 

A few 

Continuous A/B testing will allow for iterative improvements in materials and activities 

A few 

MOOCs will incorporate computer grading of open-ended assignments 

A few 

MOOCs will be available on mobile devices 

A couple 

How MOOCs could affect the economic model of higher education 

Students will cobble together their own certifications 

Many 

Meeting demand for higher education in rapidly developing countries 

A few 

Faculty members as free agents 

A couple 

MOOCs-lnside platform and course markets 

A couple 

Replacing the course catalog and course "shopping" 

One 

How MOOCs can facilitate relationships among institutions 

Cross-institutional collaborations 

Many 

Life-long connections to alumni 

A few 

Community outreach 

A couple 
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Recommendations 

For college and university administrators 

Prior to embarking on MOOC-related initiatives, institutions should carefully consider their goals and 
whether MOOCs present a realistic and financially justifiable means to achieve them compared with 
alternative strategies. Administrators at academic institutions should work with representative faculty 
members to establish a strategy for engagement, and, where agreement is reached to move forward, a 
process devised for production or adoption of MOOCs. Data collection protocols should be devised up 
front to measure relevant indicators both prior to and post MOOC engagement in order to allow for an 
objective assessment of whether stated goals are being met. By tracking personnel time commitments 
with respect to MOOC-related efforts and accounting for displacement of other productive activities, 
such as teaching or grant writing, costs of such initiatives can be estimated and compared with the 
evidence of goal achievement to judge whether MOOCs are a worthwhile investment of resources. 

To broaden access to less educated audiences, institutions must identify multiple channels of 
communication to reach potential recruits, for example, social media networks, high schools, 
employment agencies, and community organizations. Additionally, courses must be designed to serve 
less self-directed learners by incorporating motivational features and optional instructional scaffolding 
that can address differences in participant preparation levels. 

Institutions using MOOCs to build or maintain brand must manage the potentially contradictory 
objectives of increasing access to their educational offerings and building brand by carefully selecting 
where increased access will be granted, and differentiating these offerings from their core business. To 
evaluate the impact of MOOC initiatives on these goals, institutions must collect and compare pre- and 
post-MOOC metrics on student recruitment and enrollment, faculty recruitment and retention, breadth 
and generosity of donors, and quantity and size of grant revenues. Costs of MOOC production and 
delivery should be compared to other strategies known to be effective at building and maintaining 
brand or increasing access in order to ascertain which alternatives are most cost-effective. 

To reduce the costs of higher education, MOOCs could be used to substitute certain courses across 
multiple campuses, for example, those that are fairly standard across campuses; courses that serve as 
continuing professional development for working adults or that fulfill ongoing professional certification 
requirements; and courses that help students place out of developmental education. Reduction of costs 
could also be realized if academic institutions are willing to shift more of the responsibility for 
instructional support away from tenured faculty and onto non-tenured instructors, teaching assistants 
(TAs), or outsourced personnel. Negotiations with unions must address issues of permissible class size 
and related instructor income, teacher workload, and assistance from TAs. 

To increase revenues from MOOCs, institutions must find ways to confer economic value on MOOC 
completion, for example, in the form of employer-recognized credentials. Careful consideration must be 
given to the targeted audience and types of offerings so as not to cannibalize existing programs. 

To maximize the potential for MOOCs to lead to improved educational outcomes through use of 
participant data, institutions should work towards standardization of data formats across delivery 
platforms and more uniform interpretation of regulations regarding data use for research and 
evaluation. To advance the possibility of developing adaptive and personalized learning, inter- 
disciplinary and cross-institutional collaborations must be established among content experts, 
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instructors, researchers, instructional designers, programmers, and for ongoing financial sustainability, 
perhaps also for-profit content providers who have the capital to invest in this endeavor. 

For evaluators/researchers/research funders 

As a first step, it would be useful for a group of researchers and practitioners who are familiar with 
evidence-based best practices in online learning to recommend pedagogical strategies that can be 
usefully migrated to MOOCs. Conversely, MOOC researchers could benefit from soliciting the 
involvement of experienced online learning practitioners or researchers as they plan studies to 
experiment with MOOCs. Beyond attempting to improve engagement and completion of MOOCs, future 
research needs to assess the impact of MOOCs on educational and professional outcomes, for example, 
through: 

• Pre-and post-assessment of skills and knowledge, for example, through performance-based 
assessments 

• Development of metrics to assess gain in cognitive and non-cognitive skills that can be applied 
outside of the MOOC environment 

• Comparison of skill or knowledge acquisition through MOOCs vs. regular online or face-to-face 
courses 

• Follow-up of post-MOOC outcomes such as sequential learning experiences or employment 
opportunities gained 

• Broadening the types of learners represented in studies of MOOC activity and impact in order to 
avoid the presentation of results that are not applicable to the majority of learners. 

To assess the sustainability of MOOCs and to help answer the question of whether MOOCs are a cost- 
effective means to deliver education, researchers and evaluators need to: 

• Document how much costs fall for repeat offerings of MOOCs, how enrollment numbers change, 
and what the uptake rate is for any revenue-generating services 

• Compare the costs of MOOCs to the costs of alternative delivery mechanisms, as well as the 
effectiveness of each alternative in addressing a common outcome of interest, such as 
increasing participants' knowledge or skills. 

For policymakers 

MOOCs introduce a number of problems and possibilities that may not be resolved or realized without 
policy attention or pressure, in some cases because they may not be in the interests of the institutions 
which currently maintain control over the issues. Issues that must be addressed include: 

• Establishing an accreditation system to evaluate MOOCs and other non-degree-based learning 
experiences to allow learners to accumulate a portfolio of credentials that serve as a viable 
supplement or alternative to a college degree 

• Creating pathways for accredited MOOCs to be accepted for credit in higher education, or to 
satisfy government-mandated continuing education for professionals 

• Conferring the ability to use public funds towards covering the costs of such credentials 

• Clarifying what level of student services institutions must provide to MOOC participants 

• Establishing how state funding applies to colleges offering MOOCs 

• Applicability of FERPA to MOOC participants and data sharing among institutions 
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• Facilitating data format standardization across online learning platforms 

• Permitting cross-institutional Institutional Review Board agreements 

• In K-12, determining whether MOOCs can be completed for credit by high school students 
within and across states 

• In K-12, determining under what circumstances teachers can offer MOOCs, for example, how 
cross-state teacher certifications will be managed; who will pay for the course content and for 
the enrollees. 
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Introduction 


Over the past few years, observers of higher education have speculated about dramatic changes that 
must occur to accommodate more learners at lower costs and to facilitate a shift away from the 
accumulation of knowledge to the acquisition of a variety of cognitive and non-cognitive skills (see, for 
example, Dede, Ed., 2013) . All scenarios feature a major role for technology and online learning. 

Online education at the college level has been expanding rapidly over the last decade with students 
participating in single courses or even earning entire degrees without setting foot in a brick-and-mortar 
institution. According to Allen and Seaman (2013, 2014), the online enrollment growth rate in the 
United States (U.S.) has ranged between 6.1% and 36.5% in each year since 2002. Means, Bakia, and 
Murphy (2014) outline and document four major trends in how universities are using online learning: 
"self-paced, adaptive instruction and competency-based learning; blended learning; learning analytics; 
and MOOCs" (p. 46). The arrival of massive open online courses (MOOCs), which allow hundreds of 
thousands of students to participate simultaneously in a course, and are free and open to any interested 
participant, constitute a phenomenon that extends pre-existing initiatives to provide free, educational 
resources online, such as MIT OpenCourseWare (OCW), Stanford Engineering Everywhere (SEE), and 
Khan Academy. 

While the Open University in the United Kingdom has been providing distance education to large 
numbers of participants since 1971, and the Open Learning Initiative (OLI) at Carnegie Mellon University 
has been offering free, college-level courses online since 2002, the term "MOOC" was only coined in 
2008 (Cormier & Siemens, 2010). Derived from "massively multiplayer online role playing games" 
(MMORPGs), the term was first used to describe a twelve-week online course offered by George 
Siemens and Stephen Downes, Connectivism and Connected Knowledge, at the University of Manitoba, 
Canada, to an audience of 25 students for credit and an additional 2,300 students who participated 
without paying and without receiving credit (Downes, 2008). In 2011, MOOCs made frequent headline 
news after an artificial intelligence course offered by Stanford University professors attracted 160,000 
enrollees ( Markoff, 2011; Waldrop, 2013 ). The structure and pedagogical philosophy of this course was 
quite different from the connectivist MOOCs offered by Siemens, Downes, and other Canadian 
instructors. To differentiate between the two educational approaches, the terms "cMOOC" and 
"xMOOC" were coined, "c" denoting the focus on connectivism and "x" denoting "exponential," to 
describe the massive participation, or denoting "extension," for example, HarvardX as an extension of 
the Harvard campus, and MITx as an extension of MIT (see Downes' blog post, April 9 th , 2013). Most of 
the MOOCs created at present are xMOOCs. 

Currently, the MOOC concept is so new that there has been little agreement as to what actually 
constitutes a MOOC and what educational or other objectives they can and should address. Norton, 
Sonneman, and McGannon (2013), Daniel (2012), Masters (2011), and deWaard et al. (2011) offered 
some of the earliest perspectives on these questions based on reviews of the early MOOC producers, 
available writings (mostly online or journalistic), and specific cases of MOOCs. The two sections of this 
report that present Origins of the cMOOC and Origins of the Modern MOOC (xMOOC), as recounted by 
the original creators of these courses, illustrate radical differences in goals and structure of these 
learning experiences, with the only commonalities being that they are scalable and technology-based. 
But, as observed by Lewin (2013), one universal impact of MOOCs is clear: "The intense publicity about 
MOOCs has nudged almost every university toward developing an Internet strategy." 
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Allen and Seaman (2014) report that, in 2013, five percent of 2,831 U.S. institutions responding to an 
annual survey about online learning were offering a MOOC, nine percent were planning to do so, and 
fifty-three percent were undecided as to whether to engage in this innovation. Larger institutions were 
more likely to be offering a MOOC than smaller ones, as were doctoral/research institutions compared 
with institutions offering less advanced degrees. The value and purposes for engaging with MOOCs have 
been less clear, and many undecided institutions are struggling with whether and how to join the race to 
provide education using this relatively untested method of pedagogical delivery. Some faculty members 
have objected to the introduction of MOOCs on the basis that they are costly and may stifle intellectual 
inquiry if all students are exposed to the same content and educational philosophy delivered by a small 
number of "master teachers" (Kolowich, 2013a, 2013b). Allen and Seaman (2014) indicate that the most 
common objectives for the MOOCs offered by 140 or so institutions in their sample were: increasing the 
institution's visibility (27%); student recruitment (20%); innovating pedagogy (18%); and providing 
flexible learning opportunities (17%). Other objectives mentioned by less than 10 institutions each 
included reaching new students, supplementing on-campus experiences, exploring cost reductions, 
learning about scaling, and creating revenues. Surprisingly, increasing the quality of teaching and 
learning did not appear to be a priority. 

Peer-reviewed publications on MOOCs began appearing in 2008, (see review by Liyanagunawardena, 
Adams, & Williams, 2013), and some early indicators of student performance in MOOCs are emerging 
(e.g., Firmin, Schiorring, Whitmer, Willett, & Sujitparapitaya, 2013; Breslow et al., 2013; Champaign et 
al., 2014) but there is a dearth of rigorous studies investigating the effectiveness of MOOCs in 
addressing educational objectives. Researchers are still formulating ideas about how to assess 
effectiveness of different types of MOOCs. The Department for Business, Innovation, and Skills in the 
United Kingdom conducted a review of literature on MOOCs and other forms of online distance 
education (BIS, 2013), and uncovered two conflicting strands of discourse amongst learning 
practitioners. One strand focuses on "access, empowerment, relationship building and community" (p.4) 
and the potential for MOOCs to disrupt the traditional higher education system. The other strand argues 
that many of the benefits of MOOCs have already been realized in online distance education while 
challenges such as "quality of learning, accreditation, pedagogy, poor engagement of weaker learners, 
exclusion of learners without specific networking skills" (p.4) are not being adequately addressed by the 
MOOC format of delivery. 

Swan, Bogle, Day, and Matthews (2014) suggest a need to distinguish MOOCs according to pedagogical 
style as a first step before conducting research and evaluation of learning. They have created an 
instrument "Assessing MOOC Pedagogies (AMP)" that can be used to categorize MOOCs based on the 
various learning design strategies they employ. Swan et al. plan to use AMP to "characterize classes of 
MOOC pedagogies" (p.8) that can be linked to specific student outcomes such as retention and learning 
for particular audiences in various subject areas. Other researchers (e.g., Grover, Franz, Schneider, & 

Pea, 2013; DeBoer, Ho, Stump, & Breslow, 2014) point out that current discussions about the 
effectiveness of MOOCs are based on assumptions and outcome variables of previous learning 
environments and that there is a need to reframe the discussions and re-conceptualize the variables. It 
appears that few MOOC offerors are willing to address the apparently straightforward question of 
whether MOOC participants gain skills and knowledge as a result of their engagement in the courses. 
Part of this reluctance may stem from the fact that MOOCs are being pursued at many institutions for 
reasons other than improvement of teaching and learning. This being the case, alternative metrics are 
needed to assess the impact of MOOCs on these other objectives. 
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Given the ongoing alarm regarding uncontrollable costs of higher education (e.g., Bowen, 2013; Kelly & 
Carey, Eds., 2013), it would be reasonable to expect not only greater concern about the impact of 
MOOCs on the intended outcomes, educational or otherwise, but also systematic efforts to document 
the resources expended on their development and delivery. Beyond the estimates of the costs of MOOC 
production offered by the E-Learning Working Group at the University of Ottawa (see Boddy et a!., 
2013), there is little publicly available information on MOOC costs that is based on rigorous analysis. 

Lack (2013) observes that inattention to costs is pervasive in postsecondary education, and highlights 
one of the few exceptions in the field of postsecondary online learning: the National Center for 
Academic Transformation (NCAT) which helps institutions use "information technology to redesign 
learning environments to produce better learning outcomes for students at a reduced cost to the 
institution" ( NCAT) . However, the NCAT experience in itself offers a cautionary tale regarding the 
resistance of many colleges and universities to adopting technology-based innovations that apparently 
improve student outcomes, and undoubtedly reduce costs (see Miller, 2010). Carey (2010) attributes 
this seemingly irrational behavior to a lack of accountability for student learning and a lack of incentives 
to save money. Ruth (2013) explores the question of whether MOOCs can be used to help reduce 
college tuition and concludes that MOOCs may only contribute to lowering costs of higher education if 
combined with a reduction in labor costs, as experienced in successful implementations of NCAT's 
course re-design model. 

Hoxby (2014), at Stanford University, assesses the economic value of MOOCs and concludes that while 
MOOCs may have some value for non-selective postsecondary institutions, they make little sense for 
highly selective postsecondary institutions. She argues that MOOCs, which she characterizes as being 
non-selective, offering standardized content and assessments, and little student-instructor or student- 
student interaction, could substitute existing online and on-campus courses offered by non-selective 
institutions because, based on data from various national surveys, she surmises that the characteristics 
of courses at non-selective institutions are not so different from MOOCs. She does, however, question 
the assumption that cost reductions, via economies of scale, will be realized through MOOCs because 
she expects that the most popular MOOC instructors will eventually need to be paid high salaries. At 
highly selective institutions, Hoxby perceives a few opportunities for MOOCs to replace courses that 
cover standard material, such as calculus or statistics. She asserts that, otherwise, MOOCs are 
incompatible with the typical characteristics of most courses at highly selective institutions, which often 
include aspects of the instructor's research and provide relatively frequent opportunities for intellectual 
exchange. She notes that if faculty members share cutting-edge research through MOOCs, they risk the 
loss of intellectual property because of the size and anonymity of enrollments. She also suggests that it 
is risky for highly selective institutions to give away costly content for free, as this strategy is only likely 
to be successful for an "infant industry" under conditions of a natural monopoly, which clearly do not 
exist for MOOCs. Furthermore, Hoxby warns of the possibility that large investments in MOOCs may 
alienate existing students upon whom highly selective institutions rely for future donations. Hoxby 
proposes an alternative model for online education at highly selective institutions in which these 
institutions offer online courses for fee and for credit, but only to students from their own or other 
similarly selective institutions. This dual-tiered approach to delivering online education, while perhaps 
economically sound, is far removed from the movement to "democratize" education being led by other 
Stanford professors. 

It is curious that MOOCs have taken hold without much evidence as to whether they are effective in 
improving participant skills and knowledge, or in addressing other objectives, and without an idea of 
their economic value or resource requirements. As Means et al. (2014) observe, "Both irrational 
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exuberance and deep-seated fear concerning online learning are running high" (p.42). If decision-makers 
are to make rational decisions about engaging in MOOC production, it is critical to know whether 
MOOCs are both effective and cost-effective in delivering quality education or related outcomes. 

In this report we set out to provide clarity on a number of issues surrounding MOOCs and the purposes 
they serve. We queried a broad range of individuals knowledgeable about online learning, higher 
education, and MOOCs to establish what constitutes a MOOC, and to determine the goals of institutions 
and instructors offering MOOCs. After a brief review of our methods, we first present a detailed account 
of how interviewees, including administrators, faculty members, researchers, online platform providers, 
and other actors in and around the MOOCspace, define a MOOC. We describe characteristics of MOOCs 
and MOOC derivatives, and include accounts of the origins of the two major categories of MOOCs: 
cMOOCs (from the perspectives of George Siemens and Stephen Downes) and xMOOCs (article 
contributed by Andrew Ng and Jennifer Widom). Subsequently, we provide a brief summary of 
institutional approaches to MOOCs and a review of each of six major goals being pursued through 
MOOC initiatives, including whatever evidence we could glean regarding whether and how each goal is 
being measured and met. We include a number of case studies to provide detail on the MOOC-related 
activities of a few institutions in pursuit of these goals. 

Later sections of the report provide information on examples of research being conducted with MOOCs; 
the types of data available from MOOCs; the challenges of using these data for data mining and learning 
analytics; and the resource requirements and costs involved in the production and delivery of MOOCs. 
We present the results of cost analyses we conducted of MOOC development and delivery at four 
different institutions. We also present a compilation of ideas from our interviewees about how the 
MOOC phenomenon might evolve over the next five years, and their potential impact on higher 
education. We end with a few conclusions and recommendations with regard to maximizing the 
potential contributions of MOOCs to the educational endeavor. Throughout the report we use "Boxes" 
to help explain or elaborate upon specific ideas or concepts. Appendices include a summary table 
showing the types of institutions and professional roles represented in our study sample; our interview 
protocol; a brief account of what each institution in our sample is doing in the MOOCspace; a detailed 
cost analysis of online video production; sample pre- and post-surveys used to collect data from MOOC 
participants (courtesy of American Museum of Natural History and Penn State University); and a list of 
our interviewees. By providing this extensive array of information about MOOCs, we hope to improve 
transparency of the MOOCspace and to bring greater rationality to MOOC-related decision-making. 
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Methods 


Using a methodology similar to that employed by Bacow, Bowen, Guthrie, Lack, and Long (2012) to 
investigate barriers to online learning in higher education, we conducted a qualitative study comprising 
interviews with 83 individuals across a range of institutions. With the aim of eliciting perspectives on 
MOOCs from several angles, we sought out both public and private institutions of higher education, 
researchers, online learning platform providers, other for-profit education companies, and several 
additional stakeholders in the online learning space. Figure 1 and Table 1 show the distribution of 
interviewees across institutional type. Further detail on the MOOC-related activity of each institution in 
our sample and interviewee roles is provided in Appendix III, except where anonymity was requested. 
Appendix VI lists alphabetically by last name the interviewees who agreed to be identified. Figure 2 
shows the distribution of interviewee roles and further detail is available in Appendix I . Fifty-one 
interviewees were administrators, faculty members, or both at public or private institutions of higher 
education, and thirteen were researchers. Throughout the report, we refer to interviewees as each 
individual specified on their Informed Consent forms, respecting several requests for partial or full 
anonymity. 

Interviewees were identified from the existing academic and journalistic literature on MOOCs, by 
reviewing the names of conference presenters and panelists for sessions on MOOCs or online learning in 
higher education, by researching the MOOC activities of institutions on the Internet, or by consulting 
with known experts in the field of educational technology. Additionally, many of our interviewees 
suggested other people for us to interview either at their own institutions or elsewhere. We contacted 

by e-mail individuals who appeared to be 
knowledgeable about MOOCs or online 
learning based on their position in 
deciding whether and how to participate 
in the MOOCspace, their experience 
teaching or planning MOOCs, or their 
writings and research in this area. 

We contacted a total of 100 individuals on 
a rolling basis at 66 different institutions, 
39 of which were colleges or universities. 
Of the 100 contacted, 83 people at 62 
institutions were successfully interviewed 
within the time-frame of our study. At 
most institutions we interviewed one 
person, but at a few institutions we 
interviewed two or more. Most 
interviewees were based in the U.S., but 
two were in China, two in the United 
Kingdom, and several were in Canada. 
Seven of the people we contacted 
recommended we interview another individual at the same organization instead, six of whom accepted 
the invitation. Five people we e-mailed did not respond, but in each case we were able to identify 
another individual at the same institution who agreed to participate. Three people responded to our e- 
mails but did not schedule an interview time during the study period. At least one person from every 
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college or university we contacted agreed to participate. Only two individuals from the same for-profit 
institution outright refused participation. 

Figure 2: Roles of Interviewees 



■ Administrators/executives 

■ Faculty members 

■ Administrators/faculty members 

■ Researchers 

■ Other roles 


Interviews were conducted between June 2013 and February 2014 and follow-ups by e-mail with 
interviewees to obtain updates and to verify information continued until time of writing. Slightly under 
half of the interviews were conducted face-to-face and the others were conducted by telephone or 
Skype. Interviews lasted an average of 75 minutes and followed a semi-structured interview protocol 
which appears in Appendix II . Interview questions addressed what the interviewee's institution is 
currently doing with respect to MOOCs; institutional goals for pursuing MOOCs; the interviewee's role in 
the process; how the interviewee and his or her institution define a MOOC; the characteristics of 
MOOCs, including structure, purpose, enrollment level, fees, and credentials offered; how any 
credentials differ from regular course credentials; educational objectives of specific MOOCs being 
offered; educational outcomes being measured; data being collected pre-, post- and during MOOC 
delivery; how MOOC-based research is being used to improve pedagogy on-campus; personnel and 
other resource requirements; cost drivers; how costs of MOOC production compare with face-to-face 
courses; how the interviewee sees the MOOC phenomenon developing over the next five years, and 
how this might help the institution meet its ongoing goals. In some instances, not all questions were 
relevant to the interviewee. For example, researchers were unlikely to be producing or delivering 
MOOCs themselves. 

Most interviewees granted permission for the interview to be recorded, and the resulting digital audio- 
files were subsequently transcribed. All hand-written interview notes and transcriptions were coded 
using NVivo software using themes initially derived from the protocol and iteratively refined as more 
granular topics were identified. Interviewee perspectives were supplemented with a review of the 
MOOC-related literature and perspectives offered by presenters at various academically oriented, 
MOOC-related events: the " moocshop " at the 2013 Artificial Intelligence in Education conference; the 
2013 MIT Learning International Networks Consortium conference; the 2013 MOOC Research Initiative 1 ; 


1 Attendance for one author at this conference was sponsored by the Bill & Melinda Gates Foundation, for which 
we are grateful. 
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and a 2013 workshop entitled "Will MOOCs Pass the Test? Evaluating These New Online Courses," 
sponsored by American Institutes for Research and the American Council on Education. 


Table 1: Institutional Affiliations of Interviewees 


Type of institution 

Number of institutions 
represented* 

Number of 
Interviewees 

Public universities 

16 

20 

Private Universities 

14 

26 

Community Colleges 

9 

10 

Platform Providers 

5 

6 

Research organizations 

7 

8 

Other for-profit education companies 

4 

5 

Other institutions** 

7 

8 

Total 

62 

83 


* Note that while only one person was interviewed at the majority of institutions, at a few institutions several 
individuals were interviewed, for example, to include one or two administrators, one or two faculty members, 
and one or two researchers 

**Other institutions: one museum (two interviewees); one K-12 school district; one educational technology 
advocacy group; one higher education association, one venture capital firm; one private foundation; one 
independent consultant. 

Cost analyses reported in the section Resource Requirements and Costs of MOOCs were conducted 
using the ingredients method (Levin & McEwan, 2001) and were executed using the CBCSE Cost Tool Kit. 
We focused on estimating personnel costs incurred in the production of a variety of MOOCs and 
followed Levin and McEwan's guideline that personnel costs typically account for 75% of total costs of 
educational interventions (Levin & McEwan, 2001, p.53). Wherever possible, instead of using actual 
salary levels of personnel at each specific institution, we use national average salaries to calculate the 
costs of replicating production of these MOOCs. This approach not only respects the privacy of the 
individuals involved, but, more practically, allows for a comparison of the costs across a number of 
institutions without introducing local pricing influences. National average prices and benefits rates were 
obtained from the CBCSE Database of Educational Resource Prices . 

A draft of this report was circulated to all 83 interviewees prior to finalization to provide an opportunity 
to suggest edits, provide updates, and verify information. 
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What is a MOOC? 
Definitions and Characteristics 


In this section we first summarize our interviewees' definitions of a MOOC, starting with their 
interpretations of each of the four components of the name: massive, open, online, and course. We 
proceed to review a number of MOOC characteristics including: course structure; populations served; 
enrollments and completion rates; credentials and fees; and platforms used. Finally, we summarize a 
few of the course structures derived from MOOCs. 

Definition of “MOOC” 

The term "MOOC" (Massive Open Online Course) was coined by David Cormier in 2008 (Cormier & 
Siemens, 2010) to describe a twelve-week online course, Connectivism and Connected Knowledge, 
designed by George Siemens and Stephen Downes and offered at the University of Manitoba, Canada, in 
Fall semester 2008. Twenty-five matriculated students took the course for fee and for credit while 
another 2,300 participated as "open" enrollees. "MOOC" is a play on the acronym MMORPGs: 
massively multiplayer online role playing games. The section Origins of the cMOOC describes the 
pedagogical philosophy and strategies behind the development of the first "connectivist" MOOC from 
the perspectives of George Siemens and Stephen Downes (see page 31) . The key goal was "for people to 
experience what it means to be part of a social, technical system of learning where the teacher's voice is 
not an essential hub but, instead, a node in an overall network” (Siemens, formerly at Athabasca 
University). 

The structure and pedagogical philosophy of the open online courses offered at Stanford University in 
2011 were quite different from the connectivist MOOCs. The article Origins of the Modern MOOC 
(xMOOC), contributed by Andrew Ng and Jennifer Widom, describes the sequence of events leading to 
the advent of the Stanford courses (see page 34 ). Whereas the cMOOCs facilitate learning through 
participant interactions with a network of individuals and encourage participants to create, share, and 
build upon each other's artifacts (e.g., videos, blog posts), the Stanford-style MOOCs are primarily 
designed to deliver education at scale and involve more structured and sequenced direct transmission of 
knowledge. To differentiate between the two educational approaches, the terms "cMOOC" and 
"xMOOC" were coined, c denoting the focus on connectivism and x denoting exponential, focusing on 
the massive enrollments, or extension, for example, HarvardX being an extension of the Harvard 
University campus. 

The acronym "MOOC" has since been applied to a variety of online and blended courses. As Joseph 
Ugoretz, Associate Dean of Teaching, Learning and Technology, at Macaulay Honors College (CUNY) 
points out, we are "...at a place where almost any kind of online learning is called a MOOC, or if it's not 
called a MOOC, no one pays any attention to it...." Ugoretz further observes that the hype surrounding 
MOOCs has led to catchy acronyms being invented for course offerings that are similar to types of online 
courses that have existed for some years. These MOOC derivatives include the SPOC, the POOC, the 
DOCC, and the SMOC (see MOOC Derivatives below). 

"Massive” 

Interviewees expressed their understanding of "massive" in terms of actual numbers of course 
participants, the capacity for courses to enroll large numbers, or the capacity to obtain vast quantities of 
participant activity and performance data. Interviewees who suggested actual numbers of participant 

Hollands & Tirthali: MOOCs: Expectations and Reality 


May 2014 


25 


enrollments mentioned numbers ranging from a few hundred to one hundred thousand, with 500 and 
1,000 being popular benchmarks. A few felt that "massive" could apply to any class size that is too large 
to be accommodated in a face-to-face learning situation such as a traditional lecture hall or an 
auditorium. Others differentiated between MOOCs and typical online courses not only in terms of class 
size, but also with respect to instructional design of the course. Given the expectation of peer learning in 
MOOCs, a certain critical mass is necessary for the instructional design to be successful. For example, 
George Siemens defined massive as: 

Anything that is large enough that you can get sub-clusters of self-organized interests. Three 
hundred plus students could be one benchmark; another could be Robin Dunbar's number of 150 
people, which is the maximum after which the group starts to create smaller fractions. (Siemens, 
formerly at Athabasca University). 

Some interviewees focused more on the potential for courses to be massive as opposed to the actual 
numbers of enrollments. For example, they should be scalable at negligible marginal cost, and 
incorporate technological and pedagogical features that can support an infinite number of participants. 
A few interviewees suggested that the most significant contribution of larger class size in a MOOC is the 
"massive amounts of data that can be generated by these technologies." (Professor, East Coast 
University). 

"Open” 

The most generic interpretation of the word "open" with respect to a MOOC was the possibility for 
anyone with an adequate Internet connection to participate in the course. Many interviewees also 
interpreted open as free, that is, no fee is required for participation. However, most of these 
interviewees anticipated the imminent arrival of optional for-fee services such as credentialing or 
tutoring. Provided there is a basic enrollment option that remains free, interviewees will continue to 
perceive the courses as open, and massive enrollments are expected to persist. Some interviewees 
pointed to different interpretations of open as applied to access, content, and the online platform as 
described in Box 1. 


Box 1: 

What Does it Mean for a Course to be "Open?" 

Open access 

Interviewees presented different ideas about what constitutes open access to a MOOC. Karen Cator, 
President and CEO of Digital Promise, asserted that, to be considered open, a course should not 
require prerequisites or pre-tests that will limit participation. Judy Baker, Ph.D., Dean of Foothill 
College, CA, does not consider MOOCs to be open if participants must register to access the content. 
A few interviewees believe that courses are only truly open if they allow enrollment and participation 
at any time, that is, participation is not confined to the typical duration of a course. These 
interviewees were likely to be more expansive in their definition of what constituted a MOOC. For 
example, Elle Wang, a MOOC Researcher at Teachers College, Columbia University, included Khan 
Academy videos in her definition as they are more open than the typical xMOOC content that is only 
sometimes searchable on YouTube, and often inaccessible after the conclusion of the course. Others 
were more restrictive in their interpretation and felt that enrollment could occur any time within a 
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given window. 

These different understandings of openness have implications for the instructional design of MOOCs. 
For example, in order to help participants succeed, a MOOC without prerequisites may need to 
include additional scaffolding and links to self-study material for those who enroll without any 
relevant background in the topic area. If participants are not required to register, the course platform 
cannot track student progress. Absence of such longitudinal participant data precludes the 
development of adaptive learning mechanisms and inhibits the collection of learner activity and 
performance data that might be useful to researchers. 

Courses that are open to enrollment and participation at any time must be perpetually hosted and at 
least minimally monitored by the offering institutions. Participants can learn what they want, when 
they want, without the constraints of time or course structure. Such rolling course enrollment and 
completion is likely to render the experience completely asynchronous for most participants. Dr. 
Glenn Kleiman, Executive Director, Friday Institute for Educational Innovation, North Carolina State 
University, cautions against making the structure of an online course overly flexible. He believes that 
some structure is needed to facilitate professional exchanges that will occur only if participants are 
progressing roughly at the same pace through the course. 

Open content 

Some interviewees who had been previously involved with the Open Educational Resources (OER) 
movement objected to the use of "open" with respect to MOOCs because the meaning is not usually 
consistent with the "openness" of OER. Open content in the OER context implies legally open 
licensing, for example, with a Creative Commons (CC) license. Such open content is available for 
anyone to view at any time, to download, use, share, and "mash up" with their own materials. In 
cMOOCs, openness extends to participants being able to contribute resources and provide direction 
to a course, which eliminates the one-way nature of many instructor-student relationships. 

T ruly open content is common in cMOOCs but less so in xMOOCs where ownership of content is often 
contentious and has to be worked out between faculty members, their employers, course platform 
providers, and would-be adopters of the MOOC content. Timothy Renick, Vice Provost and Chief 
Enrollment Officer at Georgia State University, noted that the University's contract with Coursera 
specifies that faculty members cannot direct students to use MOOC materials from other institutions' 
MOOCs as part of their courses without paying a licensing fee. Karen Vignare, Associate Provost, 
Center for Innovation in Learning, University of Maryland University College (UMUC), pointed out that 
when contracts for the delivery of MOOCs are signed between a third party platform provider and an 
institution, as opposed to directly with an instructor, the individual instructors are unable to choose 
whether their content is made open access. 

While many MOOC instructors strive to incorporate open content, it is not always feasible and some 
have good reason to remain protective of their materials. Dave Pritchard, Cecil and Ida Greene 
Professor of Physics at MIT, expressed some reservations about open content: 

The issue of the extent to which the material should be public, I feel some ambivalence on. I 
have developed many really carefully thought out problems that elicit a lot more student 
interaction than other problems. I don't want them out there with the answers so that all the 
students can just copy the answers. I'm happy to share them with other bone fide teachers. 
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The Creative Commons license does not say that you can treat teachers any different from 
students. Then that raises a huge amount of problems with, "Who's a teacher?" How do you 
establish that you're a teacher? If you're a student taking this thing, you could masquerade as 
a teacher, and then plop up all the answers on the worldwide web. How do you prevent that 
from happening? (Pritchard, MIT). 

Open platform 

A few interviewees suggested that openness in MOOCs requires an open source delivery platform. In 
April 2013, Stanford University and edX signed an agreement to collaborate on development of the 
edX platform with a caveat that the platform would be open source by June 2013. The resulting Open 
edX initiative has now made the platform code and various related tools freely available under an 
AGPL license. In September 2013, Google agreed to offer a hosted version of the Open edX platform, 
thereby making it more accessible to institutions. Open edX carries the interpretation of "open" one 
step further in allowing institutions to retain control over their content and related platform- 
generated data by creating their own instances of the Open edX platform. Candace Thille, Assistant 
Professor, Director of Open Learning Initiative, Senior Research Fellow, Office of the Vice Provost for 
Online Learning, Stanford University, is taking advantage of this capacity by working with Open edX to 
incorporate functionalities of the Open Learning Initiative (OLI) platform into the Open edX platform. 
This will allow OLI courses to be supported on the Open edX platform and render them easily 
accessible to more people. 


"Online” 

There was little disagreement in the interpretation of the term "online." Almost all interviewees stated 
that MOOCs should be available via the Internet on a variety of devices to facilitate scale and to expand 
access beyond the traditional campus. However, experimentation with the MOOC structure has led to 
variations that include offline elements. For example, some instructors are using MOOCs as resources to 
flip their on-campus classrooms and others are "wrapping" MOOCs with their own materials for use on- 
campus (see Fisher, in press). Some MOOCs have been specifically designed to cater to a small group of 
on-campus students at the same time as being accessible to a mass of outside or "open" participants. 
Examples of the latter include the first MOOC offered by Siemens and Downes at the University of 
Manitoba, and University of Oklahoma's online courses. 

Less formally, face-to-face peer-group meetings have occurred spontaneously in different parts of the 
world and some platform providers have organized face-to-face "meet-ups" to connect MOOC 
participants. Pearson VUE testing centers may soon be used to provide physical spaces for MOOC 
participants to sit for proctored exams. Interviewees generally anticipated a trend towards more 
blended applications of MOOC content with some specific ideas outlined in the section Where are 
MOOCs Going over the Next Five Years? (see page 152 ) 

"Course” 

Most participants agreed that, to be labeled a "course," MOOCs should be bounded by time, that is, 
have a beginning and an end point; should provide a coherent set of resources; and follow a sequence of 
activities organized by an instructor in order to address specific learning objectives or goals. Siemens 
noted that without the element of “eventedness," even the Internet could be conceptualized as the 
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world's first MOOC. The degree of synchronicity required for a course varied among interviewees, with 
most believing that asynchronous participation within set time periods (e.g., one to two weeks) was 
most conducive to learning in an online course. Reasons given for enforcing set start and stop points for 
a MOOC included motivating participants to engage regularly with the course content and facilitating 
discussions and peer grading. However, a few interviewees argued against such constraints, noting that 
they simply conform with seat-time requirements for the purposes of accreditation in traditional 
university or college settings. Ugoretz at Macaulay Honors College observed that such a time-bounded 
perspective can be frustrating for life-long learners. 

Some interviewees provided additional specific criteria for a "course." For example, Ted Dodds, Chief 
Information Officer and Vice President for Information Technologies at Cornell University, suggested 
that a weekend training session does not constitute a course while an eight- to ten-week learning 
experience may do so. Others insisted on the need for assessments in order to gauge whether learning 
objectives are being fulfilled. Interviewees did not believe that certificates and credentials to prove 
completion were a requirement for a learning unit to be considered a course, although many felt that 
they should be offered. 
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Characteristics of MOOCs 


Structure 

The vast majority of existing MOOCs are xMOOCs. They are structured as weekly sequences of activities, 
over as little as two weeks and as much as sixteen weeks but, most frequently between six and ten 
weeks. Instruction is provided predominantly via several short lecture videos per week, typically each 10 
minutes or less in length, sometimes supported by supplementary readings, and problem sets or other 
assignments. Videos are often punctuated every two-three minutes with automatically graded "inline" 
questions, usually multiple choice or short answer, to help participants formatively assess their own 
understanding. Assessments that count towards the participant's final score are provided, usually 
weekly, in the form of auto-graded multiple choice or short answer quizzes, peer-graded assignments, 
and some auto-grading for computer code. Online discussion forums allow participants to engage with 
each other and course facilitators for technical and instructional support, or merely to create a sense of 
community. xMOOCs are generally delivered via third party platform providers such as Coursera, edX, 
and Udacity. Most are "cohort-based" in that they are offered over a fixed period of time, with 
participants being expected to complete activities within set windows of time. These courses and their 
associated materials often become unavailable for non-registered participants not long after their 
conclusion. Some xMOOCs are self-paced, remaining open indefinitely to participants. 


Box 2: 

xMOOC vs. cMOOC Characteristics 


xMOOCs 

Pre-determined, instructor-led, structured 
and sequenced weekly activities 

Short, content-based videos, readings, 
problem sets 

Quizzes (auto-graded), peer-graded 
assessments 

Discussion forum participation optional 

Delivered via third party platform provider 
(e.g., Coursera, edX) 


cMOOCs 

"social, technical system of learning where 
the teacher's voice is not an essential hub but 
a node in an overall network" (Siemens). 
Creation/exploration of topic area in "atelier" 
environment 

Unique products created by students (blog 
posts, images, diagrams, videos) 

Discussion forums, Diigo groups, Twitter and 
other social networking are key 
Facilitator aggregates, reviews, summarizes 
and reflects on activity in daily/weekly 
newsletter 

"Boot-strapped" platform and collaboration 
tools 


Connectivist MOOCs are characterized by a more fluid structure that addresses an overarching 
instructional goal or question but is less directive with respect to process. Siemens describes the cMOOC 
experience as the exploration of a topic area and creation of artifacts in an "atelier" environment. 
Instructors may pose initial or weekly questions and challenges, and provide a variety of text-based or 
media resources. The success of the cMOOC is highly dependent on participant interaction, for example, 
via discussion forums, Diigo groups, or Twitter. Course outcomes are often unique products, such as 
blog posts, images, diagrams, or videos generated by participants using a variety of social media. The 
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role of the instructor is to act as a facilitator by aggregating, reviewing, summarizing and reflecting on 
participant activity in a daily or weekly newsletter. cMOOCs are usually delivered using easily available 
course delivery platforms, such as Blackboard Collaborate, enhanced with collaboration tools. Box 2 
xMOOC vs. cMOOC characteristics summarizes key differences between the two major types of MOOC. 

The cMOOCs we encountered among our interviewees ranged in length from four to 30 weeks. In 
several cases, the courses have featured different instructors each week leading the activity or 
discussion. For example, Stephen Downes at the National Research Council offered the cMOOC CHANGE 
which featured 26 experts over a period of 30 weeks. Downes observed that while the course stimulated 
a great deal of discussion, it "felt too long." 

Origins of the cMOOC: 

George Siemens 

“In the 90s, I was at Red River College which became the first college in Canada to go exclusively laptop. 
Use of class time changed overnight. While teachers pretty much just substituted PowerPoints and 
display cameras for overheads, chalkboards or white boards, students started working in a totally 
different way. They were downloading music, texting, instant messaging, and doing some really creative 
things that they learned through their social systems. It became evident to me that when students were 
given their own tools of control, they acted in certain ways that you weren't able to anticipate. With the 
one-to-one technology in place, I started to play with peer-to-peer systems like Groove to investigate 
how technological and social networks influence the teaching and learning process. Stephen Downes 
was similarly exploring distributed social and technological interactions. 

We shared a central vision that learners should own their own space of interaction. Our interpretation 
of openness is that, if you learn transparently, you become a teacher, because people observe your 
learning; they see what you are doing and they can emulate or adapt it. Unintentionally, you teach 
others who are at a similar stage. If you mentor a way of being, then there is more that happens in the 
learning process than the acquisition of knowledge. There is the acquisition of the ways you interact 
with peers in that space. You become something. You don't just learn the knowledge of, say, a dentist; 
you become a dentist. That includes the acquisition of a variety of mannerisms, social standards, and 
norms that might never be embedded in a curriculum. 

In 2002, I started a "non-course" with my newsletter subscribers. I would kick off the week by asking 
one question and the mandate for participants was to discuss the question throughout the week, share 
resources, detailed opinions, and so on. This was all done on Yahoo groups. At the end of the week, I 
would review all the interactions and write an article that summarized the activity and discussions. I did 
not present content; the content was the by-product of the learning experience each week. Most 
participants were in higher education, mostly faculty plus 20 or so students. 

In 2004, I wrote a paper called Connectivism, a learning theory for a digital age about how teaching with 
technology shifted the control structures: how students were able to do things for themselves that I had 
to do for them in the past and how this had an impact on the design of the learning experience. I spent 
some time giving presentations on this notion of social network learning or connectivism, but eventually 
I thought it best to have people experience what I meant instead of talking at them. First, I ran a couple 
of online conferences on Connectivism to experiment with this idea. I used the Moodle discussion forum 
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and Elluminate as the synchronous platform. Others like David Wiley and Alec Couros were doing similar 
things. In 2007, I got together with Stephen Downes at the National Research Council in Canada and we 
agreed to run an open course on connectivism for people to experience what it means to be part of a 
social, technical system of learning where the teacher's voice is not an essential hub but, instead, a node 
in an overall network. I had just developed a certificate program at the University of Manitoba called the 
Certificate in Emerging Technologies for Learning and this course became the central course in that. We 
made Connectivism and Connected Knowledge { CCK08 ) available online and announced it through 
Stephen's daily e-mail, which had about 10,000 readers, and through a regular e-mail I sent out about 
tech trends, which had about 7,000 - 8,000 readers. We had 25 students enroll for credit and we 
allowed anyone else to join for free with no credit. We ended up with around 2,300 people subscribed. 

Before the course started, we had a few online meeting sessions with instructors who had taught online 
before or who had experimented with open courses: David Wiley, who is now with Brigham Young, had 
previously done a course with about 50-60 students; and Alec Couros who had done one with about a 
hundred students. We discussed what the experience could be like and how to make sure it would work 
well. Brian Alexander and Dave Cormier observed that what we were planning was like a massive open 
online role-playing game, so that's how the term "massive open online course" got coined [see 
Cormier's blog post on this]. Since then it just became known as a "MOOC" and we have been running 
up to three MOOCs a year. In 2011, I heard about the Stanford initiative and wrote a blog post in August 
saying that Stanford is running a MOOC. Tamar Lewin from the New York Times picked it up and I had an 
interview with her. I said "That's a MOOC they're running." We had 2,300 students, but their scale 
changed everything. 

Our emphasis was that we wanted participants to create things. Our view was that you can't take 
knowledge that is in the head of a person and deposit it in the head of another person, it has to be 
generated through active means. The pedagogical model has to emphasize creation and generation. It 
was the beginning of the "remix" culture where we start something and share it, someone else takes it 
and improves it, someone else improves it even more and maybe links to it to something else. You get a 
network innovation effect going. We believe that content should be a by-product of the learning 
experience, not necessarily a prerequisite. If you externalize how you see a concept by creating an 
artifact, someone can come by and re-mash it, re-hash it, build it, improve it, mix it up or identify any 
knowledge or conceptual errors. Our core assertion was that knowledge is a networked entity and 
learning is the process of forming, pruning, developing those networks. We received criticism because 
we tried to reduce the profile of the faculty members and encourage peer networks and structures. 

We never really emphasized that all students need to participate in a central discussion forum, even 
though we had a Moodle forum. Instead, the emphasis was on each person participating in a space that 
they owned, creating a personal learning environment and a personal learning network. Students would 
submit their blogs and their RSS feeds and we would pull out the RSS feeds from all the different 
participant websites and the Moodle forum, and create a daily e-mail newsletter that aggregated 
everything. Our course was so different from the current MOOCs because we strove to drive the 
students away from our central platform whereas edX, Coursera, Udacity bring you to their own spaces 
and determine when you can no longer access the materials. 
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Origins of the cMOOC: 

Stephen Downes 

“George and I had both been working for a long time in the area of what I was calling "learning 
networks" and what he was calling "connectivism." It's basically the same concept, and it's the idea that 
learning is a network phenomenon and that the organization of learning should therefore be network 
organization. For me, the origins of learning networks are based in the works that I did in philosophy in 
the 80s and 90s - it was called "connectionism" at the time, which is a self-organizing system. The heart 
of what I do is to try to create tools and methods to help people acquire learning for themselves, rather 
than have learning provided to them by some provider or institute. 

In 2007, George had run an online connectivism conference which was pretty popular. I'd participated in 
that. The University of Manitoba asked him to teach an online course on connectivism for the Certificate 
in Adult Education, which is a Manitoba-wide certification. I'd actually done work preparing a course for 
the same certificate a number of years earlier, called Introduction to Instruction, which was one of the 
very first online, web-based courses in Canada. He asked me if I'd co-teach this with him. The objective 
of the course became for the two of us to explain what it was that we meant, what we were up to when 
we talked about connectivism. It's wonderfully recursive. 

We decided to run the course according to connectivist principles even though this created some 
challenges for a course that was intended to be part of a university certificate program. One of the 
principles of connectivism is open learning, so all the content needed to be open. Another principle is 
diversity so we set it up such that people could use whatever system that they wanted, their blogger 
system, Twitter, Facebook, et cetera. I have a tool called "gRSShopper," that I'd been developing for a 
number of years, which allowed us to set up the course as a network. We would use gRSShopper to link 
all of the content together (see Box 11) . 

We still expected 20 to 25 people, but 2,300 people signed up. Because we had set it up in this network 
structure, we didn't face 2,300 times whatever amount of time to do certain tasks. Any place where 
there would have been a bottleneck, we used the network instead of an administrative structure. For 
example, if students contribute content to a course and it's on the university website, you've got to 
check each piece of content, especially if it's an open course, to prevent them from, say, uploading 
pornography. Because people were putting the content on their own websites, it was not necessary for 
us to pre-screen it all. We posted the regular class assignments and the grading rubrics. Some people did 
them in their own blogs, but they weren't graded unless they were enrolled in the course. We actually 
made it clear: "You choose the manner in which you participate in the course. If you just want to follow 
along and read the newsletter, that's fine. If you want to set up a blog or a feed of some sort, and 
contribute, that's fine." There was also a mechanism where you could put comments directly on the 
website. It was up to people how they wanted to contribute. 
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Origins of the Modern MOOC (xMOOC): 

Article contributed by Andrew Ng and Jennifer Widom 

Andrew Ng is the Director of the Stanford Al Lab and a Co-founder of Coursera. 

Jennifer Widom is Chair of the Computer Science Department at Stanford University. 

Online education has been around for decades, with many universities offering online courses to a 
small, limited audience. What changed in 2011 was scale and availability, when Stanford University 
offered three courses free to the public, each garnering signups of about 100,000 learners or more. The 
launch of these three courses, taught by Andrew Ng, Peter Norvig, Sebastian Thrun, and Jennifer 
Widom, arguably marked the start of the modern, instructor-directed MOOC (sometimes "xMOOC"). 

Each of these MOOCs offered learners the opportunity to watch online lectures, do machine-graded 
homework, and earn a "Statement of Accomplishment" if they passed the class. Two of these courses 
(Widom's Databases course, and Ng's Machine Learning course) were taught on a platform developed 
by Ng and a team of students, which eventually became the genesis of Coursera. One course ( Artificial 
Intelligence, taught by Sebastian Thrun and Peter Norvig) was taught on a platform led by Thrun, which 
eventually became the genesis of Udacity. Since then, many other MOOC organizations, such as edX, 
FUN, FutureLearn, NovoEd, Iversity, J-MOOC, and others, have been started to offer similarly highly 
scalable online courses. 

This article describes the origins of the three 2011 Stanford MOOCs. Because these MOOCs drew 
inspiration from many sources, it is difficult to trace the origins of every idea. We will try to do so where 
possible to give due credit, but will otherwise focus on the chronology of actions, rather than ideas, in 
the drive toward highly scalable models of education. 

Community of researchers 

The modern MOOC launched rapidly into public awareness in 2011, but contrary to popular opinion, 
MOOCs were not an "overnight success": The idea of highly scalable education had taken years of 
germination, false starts, and experimentation, culminating finally in the three highly-visible Stanford 
offerings. These MOOCs drew from a wide variety of ideas developed by a community of Stanford and 
non-Stanford researchers interested in online education. Key members of the Stanford group included 
Daphne Koller, who had been experimenting with and evangelizing blended learning (the "flipped 
classroom") at Stanford for several years; John Mitchell, who led a team developing an on-campus 
learning management system; Bernd Girod, whose students had developed sophisticated lecture- 
recording technologies; as well as Dan Boneh, Steve Cooper, Tiffany Low, Jane Manning, and Roy Pea, 
who contributed significantly to early discussions. 

Most MOOC platforms today actually pursue two major online learning activities simultaneously: (i) 
offering highly scalable forms of learning to large numbers of learners, and (ii) offering blended learning 
to small on-campus classes. The origin of participation by these platforms in blended learning stems 
mostly from the work of Daphne Koller and John Mitchell, who had experimented extensively with 
"flipped classroom" teaching in Koller's on-campus course at Stanford. Outside Stanford, Ng and Thrun 
both credit Sal Khan ( khanacademy.org ) as a huge source of inspiration. Ng was also heavily influenced 
by the work of lynda.com, and the community/forum design of StackOverflow.com . 
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The drive to scale 

What made MOOCs "revolutionary" was their availability and, especially, their scalability. One of 
Stanford's first attempts to offer scalable forms of education was started by Andrew Ng in 2007 together 
with the Stanford Center for Professional Development. About ten Stanford courses were videotaped, 
and posted online together with lecture notes and self-graded homeworks. Called the SEE (Stanford 
Engineering Everywhere) project, it offered a similar experience to MIT's OpenCourseWare, except a 
driving tenet in SEE was that every course should offer a "complete course" experience: It should have a 
complete set of materials - including lectures, homework problems, and solutions - that the learners 
could work through by themselves. Even though the technology was primitive, SEE videos came to be 
viewed by millions, and their success inspired Ng and others to spend the next few years developing and 
iterating on different versions of online education technology. 





Stanford engineering everywhere 


In the intervening years after SEE was 
launched and before the 2011 MOOCs, Ng 
launched a number of experimental websites, 
all designed with the explicit goal of giving 
every learner a good experience, no matter 
how many people participated. Screenshots of 
some of these websites are included below. 
These websites experimented with ideas such 
as tablet recording, short-format videos, 
accelerated video playback, programming 
exercises, user-generated content 
(specifically, a Wiki that anyone could edit), 
and in-video quizzes as inspired by Daphne 
Koller’s blended-learning experiments. Many 
computer science faculty members 

contributed to these ideas, and it is difficult to 
Screenshot 1: Stanford Engineering Everywhere. give credit t0 them aN; but notab | e for his 

(http:/ /see.stanford.edu/) involvement was Dan Boneh, who contributed 

significantly to the design of the tablet recording setup, and even wrote code himself for 
OpenClassroom, one of Ng's early platforms (openclassroom.stanford.edu). Boneh and Jennifer Widom 
(together with Ng) were among the most ardent proponents of using Khan-style tablet recordings, and 
contributed many of the little "techniques" that are now commonplace. Bernd Girod meanwhile 
developed a platform called ClassX for do-it-yourself lecture recording, which used computer vision for 
automatic slide synchronization and tracking of the lecturer. By allowing learners to pan-and-zoom in 
the video while it was playing eliminated the need for a camera operator. Several hundred hours of 
lecture videos self-recorded by Stanford instructors were uploaded to the system between 2009 and 
2011 . 



In this period, the Stanford teams made progress but also had several false starts. We learned lessons 
about both what would and what wouldn't work. For example: 

• We tried recording lectures in a classroom, with the instructor lecturing as usual to a room full 
of students, and editing this content later to create video segments each around five to ten 
minutes long. We learned that this approach nearly always resulted in a worse learner 
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experience compared to having the instructor explicitly plan out each five to ten minute 
segment. 

• Working with a high school, Ng experimented with website features that support a team of 
three to five physically co-present students sitting at a computer working on problems together. 
This use case was very unpopular as most students preferred to watch videos by themselves so 
that they could control the pacing. Even today, we are still uncertain how best to support 
physically co-present learners watching videos. 

• During this period, we spent a significant amount of time fine-tuning the set of tools for 
inexpensively creating MOOC videos, taking inspiration from Salman Khan and from Lynda 
Weinman's work. 



►* ****»»»• md »»» 



Screenshot 2: OpenClassroom website, version 1. Designed for use within a high school class, with a 
high school teacher facilitator, (http://openclassroom.stanford.edu/ai/shcp.php) 



Screenshot 3: OpenClassroom website, version 2, with content on a variety of computer science 
subjects, (http://openclassroom.stanford.edu) 
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Screenshot 4: Machine learning (deep learning) tutorial site -- self-paced course in Wiki format. Later 
also gained community translations of content, (http://bit.ly/deeplearn) 



Screenshot 5: ClassX system by Bernd Girod. System allows pan/zoom during playback, and features 
slide synchronization, (http://classx.stanford.edu/) 

The three 2011 MOOCs 

In 2011, the three inaugural MOOCs were launched. They came to be known affectionately by students 
as ml-class, db-class, and ai-class from the website URLs (ml-class.org, db-class.org and ai-class.com); 
they were on the subjects of machine learning, databases, and artificial intelligence. 

Ai-course was the first course to be announced, and ran on a platform developed by Sebastian Thrun, 
David Stavens, Mike Sokolsky, and a few others. The course featured short-format videos, in-video 
quizzes, multiple-choice quizzes, subtitles, and subtitle translations. 

Ml-course and db-course ran on a platform developed by four students (Jiquan Ngiam, Frank Chen, 
Chuan-Yu Foo, and Yifan Mai) working with Andrew Ng. These courses featured short-format videos, 
accelerated video playback, subtitles, in-video quizzes, weekly multiple-choice quizzes, machine-graded 
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programming exercises, and a discussion forum in which learners could post and answer questions, and 
vote questions and answers up or down. Inspired by earlier work of Jeff Ullman and his Gradiance 
system, ml-class and db-class further made extensive use of randomized quizzes, and of "mastery 
learning" in which students are encouraged to attempt an assignment multiple times until they get it 
right. Ng's team developing the ml/db-class platform set "Do What's Best for Students" as a guiding 
principle, a mantra that has continued at Coursera. 
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Screenshot 6: The machine learning course (ml-class) taught by Ng in 2011. 
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Screenshot 7: The databases course (db-class) taught by Widom in 2011. 
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Launching these free online courses generated a substantial amount of policy discussion within 
Stanford. The issue of whether completion "certificates" could be offered may have sparked the most 
discussion. Ultimately, in a series of meetings with the University Registrar, Legal Office, and the School 
of Engineering Dean's office, Ng and Thrun secured permission to offer "Statements of 
Accomplishment" instead. This term has stuck, and today, over 1,000,000 Statements of 
Accomplishment have been issued across multiple platforms. 

When building out these early MOOC platforms, one lesson we learned repeatedly was that if a piece of 
software is not designed from the outset to scale to a huge number of learners, then it will probably not 
work in a MOOC. The pedagogy of effective MOOC teaching is also very different from on-campus 
instruction, and one has to constantly keep scale in mind. It isn't simply an issue of whether the 
software will stand up under the onslaught of 5,000 simultaneous users; rather, it pertains to the 
fundamental design of the platform and pedagogy. For example, in a class of 100 students, almost any 
discussion forum design will work, since each student could conceivably read what everyone else writes. 
But in a class with 100,000 students, unless the user interface is designed from the outset to handle the 
correspondingly larger volume of discussion, then there will be many repeated/redundant comments, 
making the whole discussion forum unusable. The ml-class/db-class platform modeled its discussion 
forum after StackOverflow.com, a website for discussion about software programming, which Ng had 
frequently used in his work. The ai-class platform initially did not have a discussion forum, but a group of 
students launched a separate forum site (aiqus.com), which subsequently came to be the de facto 
course forum, used for much of the course communication. Key to any scalable discussion forum is the 
ability to vote posts up or down, so that the most helpful questions and answers quickly bubble to the 
top of the list, and so that spam can be quickly rejected by the community. 

Word about the three courses in the summer of 2011 spread mostly via social media and the popular 
press. All three courses started on the same day, and ran for ten weeks. In the end, around 7,000 
learners completed db-class and earned a "Statement of Accomplishment;" 13,000 learners completed 
ml-class; and 23,000 completed ai-class. 

After the initial MOOCs 

With the successful launch of the initial MOOCs, Ng and Thrun both started to explore ways to move 
their projects forward. Ng recruited Jane Manning to Stanford to help with faculty support, and began 
recruiting a number of faculty colleagues at Stanford and at UC Berkeley to teach MOOCs. Some of the 
early Stanford faculty members recruited included Dan Boneh, who taught a very successful 
cryptography course, Daphne Koller, and Scott Klemmer, who was later instrumental in helping to 
develop a system for peer grading. Dawn Song at UC Berkeley also played a significant role in thinking 
through possible paths forward. Bernd Girod, Daphne Koller, and John Mitchell were also key players 
during this period; indeed, Girod, Koller, Mitchell, and Ng had also previously collaborated on sharing 
resources across their earlier platforms (Girod's ClassX, Mitchell's Courseware, described below, and 
Ng's OpenClassroom). Soon after, Koller joined forces with Ng and his team to form Coursera, while 
Thrun turned his project into Udacity. Today, Coursera seeks to be a platform that works with existing 
universities, while Udacity seeks to build the university of the future. Mitchell and Girod both took 
administrative leadership positions at Stanford around online education. 
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Blended learning 

Most modern MOOC platforms such as Coursera, Udacity, and edX support two related but distinct 
visions: scalable forms of education, and blended learning for on-campus teaching. The latter of these 
visions stems greatly from the work of Daphne Koller and John Mitchell at Stanford during the period 
2008-2011. 

For many years, John Mitchell had been developing Courseware, a new LMS (learning management 
system) for use at Stanford and elsewhere. Although not designed explicitly to scale, the system had 
many features that served as inspiration for the early MOOCs, including machine-graded homework, a 
discussion forum, and sophisticated instructor dashboards. 



Screenshot 8: Courseware website, built by John Mitchell and used for Koller's early blended learning 
experiments. 

Daphne Koller was able to use Mitchell's platform to experiment with her early ideas on blended 
learning. In 2010, Koller was inspired by a talk about YouTube, and realized that on-campus education 
can be more engaging and make better use of student and faculty time if students learn the core 
material online at home in video format, leaving class time to discussions or experiential activities. This 
set of ideas, now widely known as the "flipped classroom" was little known at the time. Koller started 
videotaping her own lectures and hosting them on Mitchell's LMS, in order to "flip" her own classroom. 
In creating her videos she also invented the idea of "in-video quizzes," which today remains one of the 
best-loved features of many MOOCs. Her on-campus course also included a number of auto-graded 
interactive assessments, and used Courseware's discussion forum. Koller and Mitchell worked together 
to further his platform to experiment with blended learning in on-campus settings, for example, creating 
designs for instructor dashboards to assist just-in-time teaching. Some of these features have also 
served as inspiration for various MOOCs. 

Even though blended learning hasn't received the broad popular attention that MOOCs have enjoyed, 
they remain a key motivation for universities to partner with platforms such as Coursera and edX. In 
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early 2012, when universities were joining the MOOC movement, some were joining more to 
experiment with blended learning than with MOOCs. 

Summary 

Even though the MOOC movement appears to have emerged suddenly in 2011, the developments and 
inspirations behind the movement were many years and people in the making. A huge community of 
Stanford and non-Stanford personnel contributed ideas, and there were many years of quiet 
germination and iteration before the courses and course format were ready to "go viral." Perhaps 
ironically, none of Ng, Thrun, or Widom had heard the term "MOOC" at the time they were launching 
their initial courses, nor were they aware at the time of George Siemens and Stephen Downes' 
groundbreaking work on "connectivist MOOCs" (cMOOCs). We are still uncertain who subsequently first 
used the term "MOOC" to describe these three courses. The term xMOOC has since been proposed to 
distinguish them from cMOOCs. 

This article covers only the origins of the modern MOOC at Stanford, through the end of 2011. Many, 
many people at numerous institutions have since contributed significantly to building up the MOOC 
movement, which today continues to accelerate. 

We are thrilled at the innovations and cultural changes that the three 2011 courses have sparked. 
Scalable forms of education give instructors the opportunity to have a much greater impact on the 
world than was ever possible before. We eagerly await what universities, instructors, companies, and 
others will invent in the future. 


Acknowledgments: We are grateful to Daphne Koller for her input on the evolution of blended learning 
at Stanford, and to members of the ai-class team (Sebastian Thrun, David Stavens, Mike Sokolsky and 
Peter Norvig) for discussions about their course. Bernd Girod also provided many helpful comments. 
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Populations served 


Most MOOCs have been developed from existing undergraduate level courses. They are available to any 
member of the public who is suitably motivated and informed about the availability of such 
opportunities, who has access to high bandwidth Internet, and can read and comprehend at the 
appropriate level. The vast majority of MOOCs have been offered in English but this is rapidly changing 
as foreign universities develop their own courses and the major platform providers develop partnerships 
abroad to facilitate translation of existing courses and the creation of new, local-language MOOCs. In 
practice, the majority of MOOC participants are already well-educated with at least a B.A. degree. 

A few community colleges have been creating developmental education MOOCs targeted at a different 
set of learners compared to those enrolling in the typical xMOOCs offered by elite institutions. 
Specifically, they are aiming to serve struggling students who would normally need one or more 
semesters of developmental education courses before being eligible to enroll in credit-bearing courses 
that lead to a degree. 

Some MOOC offerors are targeting distinct groups of professionals. For example, North Carolina State 
University, New York Institute of Technology, and the American Museum of Natural History have each 
created MOOCs specifically targeting K-12 teachers and administrators. edX has been developing "white 
label" courses for corporations and Executive Education courses for which they will charge $500 per 
participant. In late 2013, Udacity announced that it would shift focus to work with corporations rather 
than with institutions of higher education ( Chafkin, 2013 ). 

Enrollments and completion rates 

While xMOOC enrollments vary from a few hundred to over 150,000 per course, they regularly run into 
tens of thousands. cMOOCs have generally been smaller, with participants numbering in the hundreds 
and low thousands. However, many of our interviewees questioned the value of enrollments as a metric 
of success because of the ease of enrolling and the absence of costs or negative consequences for failing 
to participate actively. 

I don't think that the number of enrol lees means anything other than that you successfully 
advertised the course, you got somebody to click on it. I think a much better criterion is, just the 
same as for my on-campus elective class, whether students who complete the second 
assignment pass the course. In our MOOC [Mechanics Review] last summer, 75% of the people 
who attempted half of the second assignment earned certificates. My dean would be happy with 
that. (Pritchard, MIT). 

Completion rates for cMOOCs have not typically been reported because there is rarely a required set of 
activities for the open enrollees and products created are not graded in the traditional format. On the 
other hand, because xMOOCs involve a clearly defined set of activities with a precise grading structure, 
completion is easy to assess. Completion rates for courses offered by our interviewees ranged from 
around 3% to 15% of all enrollees. As an example, Pritchard's MIT Mechanics Review has been offered 
three times, twice on the LON-CAPA platform and most recently on edX. At first offering in Spring 2012, 
2,500 participants enrolled and 67 completed the course, a 3% completion rate. In Summer 2012, the 
course was advertised more selectively to attract a specific audience of teachers and 800 people 
enrolled. Certificates were issued to 117 participants, yielding a 15% completion rate, with 90% of the 
completers being teachers. The edX offering attracted 15,000 enrollees and 1,030 of them earned 
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certificates, yielding a 7% completion rate. Some courses offer verified certificates of completion, for 
which participants must pay up front. The completion rate for the small subset of participants taking up 
this service is high. For example, in three Coursera MOOCs for which we obtained the relevant data, less 
than 1% of the enrollees signed up for Signature Track, but, of these, between 90% and 100% completed 
the course. It is unclear whether there is a trend towards lower or higher enrollment or completion for 
repeat offerings of a MOOC. We heard evidence pointing in both directions, but it is likely that this is 
largely controlled by varying marketing efforts and the narrowness or breadth of the audience targeted, 
as illustrated in the Mechanics Review example. A systematic study of this question as more repeat 
offerings are made would help determine the long term stability of MOOCs and help institutions and 
investors in the platforms assess whether MOOCs are worth the up-front investment of time and other 
resources. 

Mark Lester, Head of UK Partnerships at FutureLearn, suggested that if participants could browse MOOC 
content without registering, only those genuinely interested in participating would enroll and the 
resulting completion rates would be higher. Joseph Jay Williams, Research Fellow at the Lytics Lab, 
Graduate School of Education, Office of the Vice Provost of Online Learning, Stanford University, 
suggests that inclusion of a mandatory pre-course survey would also serve to improve completion rates 
by deterring "browsers," but, as several of our other interviewees pointed out, such open exploration 
has value in and of itself. 

Credentials and fees 

For participants successfully completing a MOOC, various institutions and platform providers have 
experimented with awarding credentials such as certificates of completion, "Signature Track" 
certificates, "Verified Certificates of Achievement," Mozilla Open Badges, and American Council on 
Education (ACE) credit recommendations. Since their inception, both edX and Coursera have provided 
certificates of completion, often signed by the instructor, to MOOC participants who meet certain 
criteria such as a minimum level of performance on course assessments. This basic credentialing is 
generally free of charge while other options carry a fee. 

Signature Track: In January 2013, Coursera introduced the Signature Track service which provides 
identity verification for MOOC participants and verifiable electronic certificates for a fee of $30-$100 
(see Fain, 2013) . Anders (2014 ) reported that 200 of 584 Coursera courses offer this option, as of 
January 2014. Learners can also earn a Specialization Certificate, by completing a series of three to nine 
related Signature T rack courses and a capstone project, for fees totaling around $250-$500. Similarly, 
edX is offering Verified Certificates of Achievement for $25-$50 for participants who agree to be 
supervised via web camera, and three sequences of courses that can earn an XSeries Certificate . 
According to Agarwal, CEO of edX, participants who pay for these certificates often donate an additional 
10% to support the edX mission. 

Among the 29 MOOC-offering institutions we engaged with in this study, only two, the American 
Museum of Natural History and Vanderbilt University, had pursued the Signature Track option for one or 
more of their MOOCs by the end of 2013. Others were considering this as a next step but noted that it 
would involve solving administrative issues related to collecting revenues from non-matriculated course 
enrollees. 

Mozilla Open Badges: One innovative idea implemented by Cuyahoga Community College was that of 
using Mozilla Open Badges as part of the game mechanics in their Pre-Algebra CCC MOOC. Christina 
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Royal, Chief Academic Officer and Provost at Inver Hills Community College, believes that badges can 
help motivate students in attaining their educational goals, and could help create a better linkage 
between student preparation and employer expectations. 

American Council of Education {ACE) Credit Recommendations: Since the 1940s, ACE has been providing 
credit recommendations for formal learning that does not take place in a college or university setting by 
identifying alignments with competencies instilled via credit-bearing courses. ACE began its service with 
military education and training and has expanded its evaluation and recommendation services to 
government agencies, corporate entities, and non-accredited education providers. ACE credit 
recommendations can act as a form of transfer credit if a college or university is willing to accept them 
(see Sandeen, 2013 ). Around 2,000 institutions across the U.S. consider ACE recommendations for 
credit. 

In February 2013, Coursera announced that ACE's College Credit Recommendation Service had 
evaluated and recommended undergraduate college credit for four MOOCs: Pre-Calculus offered by 
University of California, Irvine; Introduction to Genetics and Evolution and Bioelectricity: A Quantitative 
Approach from Duke University; and Calculus: Single Variable from the University of Pennsylvania (see 
Kolowich, 2013c ). In addition, University of California, Irvine's Algebra MOOC was recommended for 
developmental math vocational credit. Participants in these MOOCs who wish to take advantage of the 
ACE credit recommendation pay Signature Track fees and must complete an online exam supervised by 
ProctorU . Coursera advertised the option to obtain ACE credit recommendations on their website and 
through e-mail announcements to the relevant MOOC participants. At least initially, uptake for this 
service has been minimal. According to Melissa Loble, formerly Associate Dean for Distance Learning at 
University of California, Irvine Extension, only 39 of 80,000 participants across the two MOOCs offered 
by the University with an option for ACE credit recommendations signed up for the service. Of these, 38 
successfully completed the MOOCs. In Fall 2013, Pre-Calculus was offered for a second time and 37 
enrollees requested ACE credit recommendations. 

Beth Smith, President of the Academic Senate for California Community Colleges, cautioned that credit 
recommendations for MOOCs, while potentially beneficial for community college students, may cause 
confusion if, for example, a student invests the time and effort to complete a MOOC which carries the 
same title as a regular course, but is only recommended for half the credits because it does not address 
equivalent content. 

Credits: Some institutions have already offered or are contemplating offering credits for MOOCs or 
MOOC-like courses, at least to a subset of participants who may, for example, already be enrolled at the 
institution. San Jose State University (SJSU) partnered with Udacity to offer several for-credit, online 
courses in Spring and Summer 2013. This "SJSU+ Augmented Online Learning Environment" pilot limited 
for-credit enrollment to 100 students per course, who each paid $150 per three- to five-credit course. 

An additional 10,000 or so participants enrolled in each course directly via the Udacity platform for free 
and without receiving credit. University of Oklahoma has developed online courses which count for 
credit for the University's own students, but are also open free and without credit to non-matriculated 
students. Matriculated students pay tuition equivalent to the on-campus courses. Georgia Tech's 
MOOC-based Online Masters of Science in Computer Science (OMSCS) program, being developed in 
partnership with Udacity, offers courses for full credit, at $134 per credit (see Case 5 ). 

A few institutions, including University of Maryland University College, Georgia State University, and 
Penn State World Campus are exploring ways to award credits to MOOC completers who are able to 
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demonstrate competencies equivalent to on-campus offerings, (see Box 3 for Georgia State University's 
approach to this). 

We look at MOOCs as a great opportunity for informal learning. If the provider, either at the 
college, or Coursera, or Udacity has gone the extra steps of having ACE accreditation reviews of 
their MOOCs, then we will transfer and accept that content, providing our student shows 
evidence of competency. (Vignare, University of Maryland University College). 

Through its MOOC2Degree initiative, Academic Partnerships is developing MOOCs with partner 
universities to serve as entry points into full online programs. Students who complete the initial MOOC 
and enroll in the full program at the partner institution can obtain credit for the MOOC towards their 
degree program, subject to criteria established by the university. (See Case 4 for more details on 
MOOC2Degree). 

Offering credits for MOOCs can be contentious because, unless the MOOC is as rigorous as an on- 
campus equivalent, there is a risk of diluting the brand of the offering institution. Furthermore, unless 
participants pay equivalent tuition, it will undercut the institution's core business. Several interviewees 
asserted that credits should not be offered for MOOCs unless they provide the same rigor in learning, 
assessment, and identifiability of participants as face-to-face courses. A few observed that rampant 
cheating in online courses serves as a major deterrent to offering credentials. 

Platforms used 

Most of the xMOOCs offered by our interviewees or their institutions were offered via the Coursera and 
edX platforms, with a small number offering courses on both platforms. Other platforms mentioned 
included Blackboard CourseSites, Blackboard Collaborate, Udacity, Desire2Learn, Google CourseBuilder, 
NextThought, LON-CAPA (Learning Online Network with Computer-Assisted Personalized Approach), and 
Canvas Network. By being selective about which institutions may deliver courses using their platforms, 
Coursera and edX have successfully leveraged existing institutional brands to create their own brands, 
which draw the attention of millions of MOOC participants. However, some faculty members and 
institutions are choosing different platforms which provide additional functionalities. For example, 
Canvas Network by Instructure provides more options for math quizzes, allowing answers in the form of 
equations in addition to the multiple choice format common to other platforms. NextThought has 
developed a sophisticated structure to facilitate peer learning in contrast to the typical rudimentary 
discussion forums. Beyond our study sample, MOOC. List provides a directory of MOOCs offered by 
around 46 different providers, with Coursera, Saylor, Canvas, and edX courses being the most 
ubiquitous. 

Instructors offering cMOOCs cobbled together their own delivery systems by combining a variety of 
tools. For example, Downes and Siemens used Blackboard Collaborate and gRSShopper (see Box 11) , a 
tool created by Downes to aggregate feeds from various social media tools. Matthew Martin, Professor 
of Communication Studies at West Virginia University, and colleagues used Drupal, an open source 
content management system, in conjunction with freely available social media resources. 
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Box 3: 

Accepting MOOCs for Credit: Georgia State University 

According to Georgia State University's Complete College Georgia, 2013 Status Report, the University is 
working with the University System of Georgia to develop "principles and procedures for the support 
and credentialing of innovative means of learning, including MOOCs." Timothy Renick, Vice Provost and 
Chief Enrollment Officer at Georgia State University, explains the thinking and practicalities behind this 
move: 

What we were saying about MOOCs is no different from an offer we make to students across the 
board, which is to say: whether you are a vet coming in from military service, an adult learner, or an AP 
student, you can make the case to us that you have certain competencies. One reason that this is 
relatively non-controversial on campus is because we are serving students that are low-income and 
struggling to pay their bills to get through college. We are trying to make it easier for them to do so. So 
MOOCs allow a less expensive and more flexible avenue to getting credit. Ninety percent of our students 
have jobs, so we want to keep open opportunities for self-paced learning and learning outside the 
classroom. 

We don't automatically take ACE recommendations, despite the fact we are trying to make things easy 
for our students. Our faculty are reluctant to turn the ability to determine what the functional 
equivalent is for a Georgia State course over to an external body. We do this all the time for things like 
AP credit, we have standard tables and if the AP people say that the student got a 4 then we give them 
college credit for that course. We are developing mechanisms of various sorts. In some cases we have 
found that there are standard, well-accepted, and well-established national exams that exist for certain 
subject matter and we do offer CLEP exams through our testing centers so students can come if they are 
claiming competencies in an area where there is a CLEP exam. 

We also have some academic departments that, long before MOOCs, developed their own measures, 
often exams, or interviews, or project presentations, but mostly exams. In cases where there is no CLEP 
exam or home-grown exam, we encourage the departments to develop a measure for these students 
and others that come afterwards. The takers of our offer to consider MOOC-based competencies have 
been few yet - just a couple - so the workload for the departments has been relatively light. But I think 
there would be a workload issue down the road if this was to become the norm and hundreds of 
students came in to say they had these competencies. We would face pressure to find a way to resource 
that because right now it is fairly intensive work to develop the instruments to measure whether the 
student has the competency. 

But I could envision a future where, if this continues, this sort of assessment becomes not unlike what 
we do with credit transfer articulation. Right now to transfer credit, especially from outside the state, is 
very work-intensive. It requires someone in the department sitting down with the syllabus of the course, 
making a judgment about how that relates to other courses the department currently offers, finding an 
equivalency, then filling out the paperwork electronically to let the central office know it has been 
approved. Then it is easy for subsequent students who take that particular course and want to transfer 
the credit to Georgia State. So something similar could happen in this space where if we develop good 
measures, we could even share those with other institutions in the state that have similar core courses. 
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At the moment, students do not have to pay for this service but we are having a senate subcommittee 
look at that. Because the space is small we haven't been pushed that way yet. They do have to pay 
something like $50 to take the CLEP exam, but we don't have a structure yet if they want to take some 
sort of graded assessment in these other areas, but we will develop some model and I expect it would 
be about the same or maybe a bit more because it has to be personalized more. If the workload 
becomes significant we'd have to look at some release time or some kind of compensation for faculty 
working on this. 

Source: Timothy Renick, Vice Provost and Chief Enrollment Officer, Georgia State University. 
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MOOC Derivatives 


As MOOC producers experiment with various design aspects of these courses, derivative course 
structures with associated acronyms have proliferated. We heard about several of these from our 
interviewees, but many others have also been written about in the popular press and many more are 
likely to arise in the future. Examples include: 

MOOC-Ed: These MOOCs are specifically targeted at teachers and offer professional development. 

SPOC: Small Private Online Course (see Fox 2013) . These closed courses implemented, for example, by 
Fox and Patterson at the University of California, Berkeley, integrate MOOC materials into an on-campus 
course, allowing local faculty to incorporate activities that enhance the learning experience for their 
students. 

"Wrapped" MOOC: Similar to SPOCs, some instructors have re-designed their courses around a MOOC 
offered by another institution, "wrapping" the MOOC with their own content, activities, readings, and 
assignments (e.g., Doug Fisher, Director of the Vanderbilt Institute for Digital Learning, Vanderbilt 
University, see Fisher, in press). A wrapped MOOC differs from a SPOC in that the former enables 
students to participate in the global learning cohort of the MOOC, as well as participating in their local 
cohort (see Fisher and Fox, 2014, p.23). In contrast to flipped classrooms, MOOC materials may be 
incorporated into class time, rather than being assigned as homework. 

"White label" MOOC: These courses offered by edX may have massive enrollments but are only available 
to employees of a particular company or to members of an organization. 

Mini-MOOC: These are online courses that are openly offered using technology and pedagogical 
features that can accommodate a large number of students, but in practice enroll only a few. An 
alternative use of the term is to describe very short, educational experiences delivered online to large 
numbers of participants, akin to a webinar. 

DOCC: Distributed Open Collaborative Course. The DOCC concept was created by FemTechNet in 
reaction to MOOCs. The DOCC involves students and instructors from multiple institutions who engage 
in networked learning through a collaborative open course (see Jaschik, 2013 and DOCC 2013) . 

Participatory meta-MOOC: In January 2014 Cathy Davidson at Duke University launched a MOOC via 
Coursera entitled History and Future of (Mostly) Higher Education. Simultaneously, three related face-to- 
face courses were offered at Duke University; University of California, Santa Barbara; and Stanford 
University. The MOOC was developed as part of the FutureEd Initiative sponsored by HASTAC, a virtual 
learning network. Davidson labeled the course a "meta-MOOC" because, among other topics, it 
addresses the possibilities and pitfalls of MOOCs in Higher Education. 

SMOC: Synchronous Massive Online Course. The SMOC was introduced by the University of Texas at 
Austin, which offered Introduction to Psychology to on-campus and non-enrolled students 
simultaneously. The instructors lectured live on the Internet twice a week. 

POOC: Personalized Open Online Course. POOCs that offer adaptive learning are more of an idea than a 
reality at present. The success of these courses will depend on collecting and processing massive 
amount of data from the students in order to personalize the learning based on individual progress. 
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How and Why are Institutions Engaging with MOOCs? 


Colleges and universities have adopted several different stances towards engaging with MOOCs. Some 
are actively developing MOOCs and may be termed "producers," some are using MOOCs developed by 
other institutions in their programs and could be termed "consumers," and a few are doing both. Others 
are adopting a "wait-and-see" approach, and some have considered MOOCs and have either decided 
against any form of official engagement, or have not met with interest from faculty members to pursue 
them. 

While many people assume that MOOC production is limited to "elite" universities, we found this not to 
be the case. While it is true that MOOCs have "catalyzed big brands into what is fundamentally a 
distance learning game" (Lester, FutureLearn), other institutions, including a few community colleges, 
have concluded that they know how to serve their particular students best, and have created MOOCs 
tailored to the needs of their own student populations. 

"Consumers" of MOOCs are integrating MOOCs created by other institutions into their course offerings 
in flipped classrooms (e.g., San Jose State University), or as wrapped MOOCs (e.g., Vanderbilt 
University), or simply as supplemental resources for their students. Instructors at the University System 
of Maryland are experimenting with a variety of approaches to embedding MOOCs created by others 
into their classes (see Griffiths, 2013; Ithaka S+R, 2013). A small number of institutions have declared 
willingness to consider MOOCs for credit (e.g., Georgia State University, University of Maryland 
University College). A few universities such as Vanderbilt University and University of California, 

Berkeley could be categorized as both producers and consumers, with some instructors creating MOOCs 
and others using MOOCs created by their own or other institutions in their classes. 

Of the 39 colleges and universities that participated in this study, 28 had already offered a MOOC or 
integrated MOOCs into their courses, or were in the midst of doing so. Most fell into the producer 
category and were developing xMOOCs, with only a small fraction offering cMOOCs. Institutional 
approaches varied from tightly coordinated and controlled, top-down efforts in which senior 
administrators hand-picked academic departments and instructors to offer MOOCs; to more democratic 
processes in which a committee solicited and reviewed proposals from faculty members; to more 
grassroots approaches where the administration took no official position but individual instructors 
developed and offered MOOCs of their own accord. In some cases the institutional approach towards 
MOOC production evolved rapidly from an ad hoc process to establishment of formalized teams 
dedicated to various functions such as videography, instructional design, course support, and 
contractual negotiations. Appendix III summarizes the MOOC-related activity of each institution we 
engaged with, except where an interviewee requested anonymity. 

Our interviewees identified a variety of institutional goals for engaging with MOOCs which we organized 
into six categories as shown in Table 2 below: extending the reach of the institution and access to 
education; building and maintaining brand; improving economics by lowering costs or increasing 
revenues; improving educational outcomes for both MOOC participants and on-campus students; 
innovation in teaching and learning; and conducting research on teaching and learning. In subsequent 
sections we address each goal in turn, describing how the goal is being addressed, any evidence of 
success, and obstacles encountered. 
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Table 2: Institutional Goals for Developing and Delivering or Using MOOCs 


Institutional 

Goal 

Number of institutions offering /using MOOCs stating this 
as a goal 

% of all 

interviewees who 
raised this as a 
goal (n=83) 

% of total 
(n=29)* 

Private 

Universities 

(n=10) 

Public 

Universities 

(n=15) 

Community 

Colleges 

(n=3) 

Extending Reach and 
Access 

65% 

7 

8 

3 

42% 

Building and Maintaining 
Brand 

41% 

3 

8 

0 

25% 

Improving Economics 

38% 

2 

8 

1 

29% 

Improving Educational 
Outcomes 

38% 

4 

5 

2 

20% 

Innovation 

38% 

3 

7 

1 

19% 

Research on Teaching and 
Learning 

28% 

4 

3 

1 

18% 


* Includes one museum in addition to the universities and colleges, explaining why two of the "% of total" 
numbers are higher than expected from counting the colleges and universities alone. 


Our findings are somewhat consistent with other investigations regarding the goals of institutions 
offering MOOCs except that in two other studies branding appeared to be more important than 
improving access to education. A 2013 study, To MOOC or not to MOOC: Strategic lessons from the 
pioneers, conducted by Cathy Sandeen, Vice President, Education Attainment and Innovation, American 
Council on Education (ACE), and Dave Jarratt, Vice President of Marketing at Inside Track, identified 
similar goals to the ones we found from a survey of administrators at nine institutions of higher 
education, and a separate survey of 108 faculty members who had offered MOOCs. According to 
Sandeen, "A common motivation was reputational, and also for outreach and dissemination of 
knowledge; and another was to provide an incubator for new ideas." These respectively fit with our 
branding, access, and innovation goal categories. Respondents in Sandeen and Jarratt's study also 
mentioned the objectives of lowering costs, increasing revenues, and improving outcomes for students, 
but they did not mention MOOCs for conducting research on teaching and learning. Sandeen and Jarratt 
did not quantify their results so we cannot compare the frequency with which goals were mentioned by 
our respective samples. 

In their 2013 annual survey of online learning in higher education, Allen and Seaman (2014) asked 
representatives of 2,831 colleges and universities in the U.S. to indicate the "Primary objective for your 
institution's MOOC." Only five percent of their respondents had offered a MOOC and, among these, the 
most common objective was to "increase institution visibility," selected by twenty-seven percent of this 
subset. The second most important objective, selected by twenty percent of the MOOC producers, was 
to "drive student recruitment." Both these objectives fit within our second most popular category of 
building and maintaining brand. We cannot directly compare frequency with which each goal was 
mentioned because Allen and Seaman's respondents were limited to a single choice of objective, while 
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ours could mention as many as they wished. Additionally, while the options offered in Allen and 
Seaman's survey covered four of the goal areas we heard about from our own interviewees, they did not 
offer choices that would correlate directly with improving educational outcomes, or conducting research 
on teaching and learning. However, it appears that branding was most important to their sample while 
increasing access and reaching a wider audience was most important to ours. Very few of their 
respondents, less than three percent, selected the objectives of reducing costs or generating revenues. 

The Alliance for Higher Education and Democracy (AHEAD) at the University of Pennsylvania recently 
conducted a poll of administrators, faculty members, and other personnel at institutions of higher 
education in the U.S and found that, among the approximately 44 respondents at institutions offering a 
MOOC, 57% strongly agreed that MOOCs may be a potentially effective mechanism for “Raising 
Institutional Profile" (AHEAD, 2014), which we would equate with branding. Fifty percent of the 
respondents strongly agreed that MOOCs can help improve access around the globe, 40% that MOOCs 
can help improve access in the U.S., 34% that MOOCs can improve pedagogy, and only 19% that they 
can reduce costs. Interestingly, and tellingly, confidence in the potential for MOOCs to reduce costs was 
higher among respondents from institutions that had not yet offered MOOCs. For all other goal options 
offered by AHEAD, MOOC offerors appeared more confident in the potential for MOOCs to achieve the 
goals than MOOC abstainers. 

Steven Mintz, Executive Director of the Institute for Transformational Learning at the University of Texas 
System, highlighted differences in goals among members of the edX consortium, noting that for some 
members the primary goal is branding, "being relevant," or helping solve the political problem that their 
enormous endowments are being spent on relatively few students, whereas "state systems like my own 
hope to use MOOCs to expand access and affordability, and to potentially use MOOCs for credit." In 
general, for less wealthy institutions, return on investment is a greater priority. Several of our 
interviewees expressed frustration at the lack of agreement across administration and faculty members 
within their own institutions regarding the goals for engaging with MOOCs. 

We heard a number of reasons to explain why some institutions are not embracing MOOCs. Some do 
not accept the notion that mass delivery of education will allow for the quality of learning that can be 
fostered on a small scale. For these institutions, quality is a priority over quantity. Others doubt that 
MOOCs can help them fulfill their mandates, for example, Royal at Inver Hills Community College 
observes that "Through the lens of innovation, MOOCs are an important initiative with great potential. 
Through the lens of college priorities such as retention, persistence, and completion, they are probably 
not." Several institutions are simply being cautious and financially circumspect, letting the early 
adopters or "MOOC pioneers" spend the time and money to figure out what works so that they can 
adopt best practices later. A few are pragmatic in the realization that their brands and MOOCs cannot 
compete with those of elite institutions. Some are reluctant to give up significant amounts of control, 
content, and data to third parties, and are either planning to figure out how to offer MOOCs themselves, 
or are waiting for alternative consortia to coalesce in which they might wield more influence. 

Other institutions lack the resources for MOOC development and are hard pressed to justify cutting back 
other initiatives in order to divert resources to MOOCs. In particular, a few question how they will be 
serving their paying students well by expending such significant amounts of time and effort to educate 
the rest of the world, especially when most MOOC participants already have a college education and 
their own students are still trying to earn one. For example, a senior administrator at a leading UK 
university remarked: "We are guite dubious about conducting expensive experiments in the area of 
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education when we have our own students paying £9,000 a year for the education we provide them, as 
opposed to learners outside the university." These concerns appear to be validated by the reports we 
heard from a few interviewees that on-campus students are already beginning to question why they are 
paying tuition to underwrite the production of MOOCs, and yet they cannot earn credit for them. 

In some instances, the college or university administration has not taken a formal position on MOOCs 
but individual faculty members, the equivalent of Moore's (2014) "technology enthusiasts," are 
experimenting on their own, either offering MOOCs they develop themselves without institutional 
resources and support, or integrating MOOC materials from other institutions into their courses. In 
these situations, the goals of individual faculty members are primarily to advertise their subjects and 
expand their reach to a wider audience. In a few cases, MOOCs serve as a source of data for research. 
Even when an institution does offer support for MOOC development, some faculty members prefer to 
act independently without being subject to institutional restrictions, and without being limited by the 
institution's technology capacity, or lack thereof. Martin at West Virginia University also suggested that 
MOOCs offer a far less restrictive outlet for faculty members to disseminate their work and expertise 
than publishing companies which are increasingly demanding rights to content. 
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Goal 1: Extending Reach and Access 


Extending the reach of the institution to a wider audience and improving access to education was the 
most commonly identified goal for offering a MOOC, mentioned by 65% of the institutions in our study 
that were offering or using MOOCs, and 42% of our interviewees overall. These interviewees included 
administrators and faculty members from seven private universities, eight public universities, three 
community colleges, and a museum. The frequency with which this goal arose is consistent with the 
message offered by each of the three most prominent MOOC platforms. Coursera, edX, and Udacity 
each highlight in their mission statements access to a world class education for all as a goal of their 
enterprises: 

Coursera is an education platform that partners with top universities and organizations 
worldwide, to offer courses online for anyone to take, for free. We envision a future where 
everyone has access to a world-class education. We aim to empower people with education that 
will improve their lives, the lives of their families, and the communities they live in. ( Coursera ). 

Through our institutional partners, the XConsortium, we present the best of higher education 
online, offering opportunity to anyone who wants to achieve, thrive, and grow. (edX). 

Our mission is to bring accessible, affordable, engaging, and highly effective higher education to 
the world. We believe that higher education is a basic human right, and we seek to empower our 
students to advance their education and careers. ( Udacity) . 

Some institutions claimed to be pursuing an altruistic goal of reaching the masses globally with high 
quality educational experiences, while others presented more defined goals of reaching a specific 
population, such as providing continuing education for alumni, or solving a particular challenge related 
to access. For example, Martin at West Virginia University, offered a cMOOC on social media with 
several colleagues from the Department of Communication Studies with one goal being to provide 
continuing education to alumni who had not had opportunities to take courses related to social media. 

Reaching the masses 

Many interviewees spoke of MOOCs as a means to fulfill an instructional or personal mission to make 
education universally available: 

The reason I am so passionate about this is because I want the American dream sustained. I 
don't see how we are going to get there unless we up the level of knowledge, skills, and expertise 
that are broadly disseminated in the general population. I see MOOCs as a vehicle to do that. 
(Professor, East Coast University). 

We want to make education available globally. Ideally all people in all places would be able to 
get a quality education. We want to provide a course of [our institution's] quality to outside 
people. (Senior Administrator, East Coast University). 

We want to provide education for free to anyone who is interested... .We like the idea of "The 
Walking Dead MOOC" because we feel it does what MOOCs initially set out to do, which is to 
provide education to anybody that could be interested at any level; and that it's open and free 
education; and that it's fun and interesting and not necessarily about credit. It's about the 
experience. So we feel that if we can just get one more person to be excited and learn something 
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about a topic they haven't known about before, we would consider it successful. (Melissa Loble, 
formerly at University of California, Irvine Extension). 

The University as a place-based institution has public engagement as part of its mission. We 
have wonderful events and speakers on campus and the ability to teach in a public mode. I think 
this is using technology in a radical way to extend that. (Harper, University of Oklahoma). 

I won't try to make the argument that MOOCs are a panacea, or applicable to all types of classes 
in all settings. However, for introductory engineering and science classes, I am comfortable 
asserting that it is an important aid in democratizing education, and particularly for our 
institution, a way for people who really want to get access to our education, to do so. 
(Development Officer, Highly Selective University). 

Part of Columbia's mission is to educate the world and spread knowledge. (Sreenivasan, formerly 
at Columbia University). 

Several instructors were motivated to offer MOOCs because of their passion for their subject and desire 
to make it accessible to the general public. 

The faculty member for the "Geospatial Analysis and Mapping" MOOC was very committed to 
having topography understood at a lay level, and why that's increasingly important in our world 
of digital positioning; why people need to understand more about maps. He had the personal 
guest of reaching thousands of people. He said at one point, "I'll be teaching in a single MOOC 
more people than I'll ever teach in my entire career. " He was very motivated by that. (Lawrence 
C. Ragan, Director, The Center for Online Innovation in Learning, Penn State World Campus). 

My goal is to teach as many people as possible about viruses, which is unusual for a 
researcher. ...Between the blog, YouTube, iTunes, and Coursera I have massive involvement of 
people. It is really important that scientists reach out to the public with what they are doing. 
(Racaniello, Columbia University). (See Case 1 ). 

Improving access to community college through developmental education MOOCs 

Community colleges have a very specific goal of improving access for students who want to attend 
college but do not have the necessary preparation in math and English. Burck Smith, CEO of 
Straighterline, an online course provider, believes that MOOCs can offer a "no-risk failure model” for 
struggling students. Instead of increasing the cost to institutions by adding more remedial services, 
MOOCs may provide a no-risk, low-cost option that might encourage and motivate some at-risk 
students. A small number of community colleges including Mt. San Jacinto Community College, CA, 

Wake Technical Community College, NC, and Cuyahoga Community College, OH, have created and 
offered developmental education MOOCs with the aim of saving students time and money by setting 
them on an accelerated path to credit-bearing courses, and more timely completion of a degree. 

Some students wanting to attend community college do not have college-level math and keep 
failing the introductory level classes. They either cannot afford to retake the classes, or are 
prohibited from doing so by the new policies. If they can take a math MOOC and help get 
themselves to college level in math, they may be more likely to succeed first time in the credit- 
bearing courses, especially as there is less pressure from being judged in a MOOC. So MOOCs can 
improve access and allow students another chance to succeed. (Online Learning Coordinator, 
Large California Community College). 
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To increase the likelihood that students would benefit from the experience, the designers of Mt. San 
Jacinto's Crafting an Effective Writer MOOC included a week-long unit on "how to be successful as an 
online learner" (Patricia James, Dean of Distance Education, Mt. San Jacinto Community College, CA) and 
made 14 people available during the course to respond to student inquiries and requests for support. 
However, assessing whether these MOOCs reach their target population or help students test out of 
developmental education has not been easy because student identities in a MOOC are difficult to match 
against on-campus identification systems, thereby making it difficult to track the students' post-MOOC 
course-taking trajectory. 

Not all interviewees were optimistic that MOOCs can help struggling community college students, with 
some pointing out that expecting students at the developmental education level to "self-remediate" 
may not be realistic. 

One of the challenges with developmental education and MOOCs is the level of guidance and 
support given. One of the lessons we learned is that even when you structure learning in a very 
methodical way, people inherently have different levels of motivation and desire to move 
forward at a certain pace. In general, we need to move towards more competency-based 
frameworks where students can move forward at their own pace as long as they complete the 
competencies, whether this takes two months or two years; and you pay tuition until you 
complete it. MOOCs have a framework that can be very helpful in supporting that, but there still 
has to be a level of guidance and support, where you just can't have one faculty member 
teaching 10,000 students and expect that there will be a level of guidance and personalization at 
where we are at today. (Royal, Inver Hills Community College). 

Beth Smith, President of the Academic Senate for California Community Colleges, notes that students 
often believe that online courses are easier than face-to-face courses, and are not adequately prepared 
to do the necessary work. She also cautions that the amount of reading required in an online course can 
be problematic for some community college students. Shanna Jaggars, Assistant Director at the 
Community College Research Center (CCRC), Teachers College, Columbia University, similarly asserted 
that, to benefit from MOOCs, students must have relevant background knowledge and specific interest 
in the subject; be self-directed and strongly motivated to learn; and have strong skills in time 
management, metacognition, and information literacy. She cautions that this constitutes a small 
proportion of all community college students, while the majority requires more structure, guidance, and 
encouragement than is presently provided by MOOCs. Past research by Jaggars and others at the CCRC 
(see Online Education and Instructional Technology papers at CCRC) has not found online learning to be 
helpful in improving graduation rates for community college students in general, but a more recent 
study by Shea and Bidjerano (2014) indicated that students taking some of their early courses online 
were more likely to obtain a community college credential than students who took classes only on- 
campus. 

MOOCs to overcome resource constraints 

In addition to the generic goal of improving access, some institutions are utilizing MOOCs to overcome 
specific constraints in order to improve access to courses and degree programs. A few interviewees 
suggested that MOOCs can alleviate current infrastructure constraints for education both domestically 
and in rapidly developing countries where the existing physical plant cannot accommodate the growing 
demand for postsecondary education. 

Hollands & Tirthali: MOOCs: Expectations and Reality 


May 2014 


55 


For the Online M.S. in Computer Science program, a primary goal is to provide access to students 
who are not able to take on-campus courses, and to accommodate people we don't have room 
for. Last year we had 1,400 applicants and over half were eligible. We could only accept about 
140, so we felt there was an underserved population and we wanted to see how big that is. 

There was some concern that the online program would decrease demand for the on-campus 
program, but, in fact, the number of applications for Fall 2014 increased by about 30%, and the 
number of new students starting this Fall will probably be at least 40% greater than the Fall 2013 
class. (White, Georgia Tech). 

I am on the Global Engineering Deans Council. In China, the number of students enrolling in 
college is going up almost 20% at a compound annual growth rate. Over 40% of these students 
are enrolling in engineering. You have all this incredible growth in the number of engineering 
students and nobody there to teach them. That is a need that MOOCs could address. (Peercy, 
University of Wisconsin-Madison). 

MOOCs can potentially ease the pressure on oversubscribed programs or courses. Catheryn Cheal, 

Ph.D., Associate Vice President and Senior Academic Technology Officer at San Jose State University, CA, 
noted that the MOOCs being developed by the university in conjunction with Udacity, such as 
Introduction to Psychology and Java Programming, are all "bottleneck" courses. Many students are 
blocked out of these required courses due to capacity constraints, and are delayed in making progress 
towards degree completion. A senior executive at a nonprofit, online learning company indicated that 
bottle-neck courses at state universities and community colleges may be the next target for MOOC 
production. 

We are looking at big growth in developmental curriculum, remedial curriculum, the lower-level 
courses that are log-jammed, that are pinch-points in the curriculum in community colleges and 
undergraduate education: statistics and pre-calculus, calculus, micro and macroeconomics - the 
courses with massive enrollments. We have started with high-end, elite courses and the next 
phase will be the courses where large, state-funded, land-grant systems cannot find space for all 
the students who need to take them to complete degrees. (Senior executive at a nonprofit, 
online learning company). 

Ann Kirschner, University Dean at Macaulay Honors College, NY, believes that MOOCs can increase 
access to instructors skilled in specialized domains and niche subjects. 

I think MOOCs will help because not all institutions can be expert in everything. So if students 
come to us to study a language or specialized part of physics for which we don't have a faculty 
member, instead of going to Stanford for a semester they can take a MOOC. If the Amazon of 
MOOCs existed, I could tell a student to study George Eliot with Professor X at Cambridge, to 
supplement resources and to develop an education collective that could only exist in a network- 
distributed 21st century world. (Kirschner, Macaulay Honors College). 

Kirschner also suggested that MOOCs can provide flexibility for students to create their own programs 
with courses from various institutions to best match their abilities and interests. Indeed, Coursera has 
recently begun offering certificate-bearing sequences of courses, and several universities offer similar 
course "series" ( Kolowich, 2014 ). MOOCs, as with online learning in general, can improve access to 
education by providing flexibility in time and place of study, either for a commuter population on- 
campus or an off-campus audience beyond city, state, or national borders. 
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I have been thinking about the best use of our students' time. Could we take some of their 
requirements and move them online to give them greater flexibility? We have the special 
pressure of a commuter population. Could we find ways to give them flexibility and opportunities 
to interact online as students which they don't have now? I am very interested in experimenting. 
(Kirschner, Macaulay Honors College). 

We are interested in growing an international audience - people who never actually come here. 
That's a target population for the museum and this could be a good way to get them in. (Kinzler, 
American Museum of Natural History). 

MOOCs are also being used to address constraints of "traditional" online courses which are often 
restricted in size to the same degree as on-campus courses so as not to overextend the instructor. 

For programs like ours that have ambitions of growing larger, that's a real concern. You don't 
have the mechanisms to scale to large enrollments when you're considering every class can only 
have 20 students. Part of our motivation for going into MOOCs was to examine how can you 
serve large enrolling classes, efficiently, effectively, but at the quality that we want to brand as a 
Penn State quality experience. We were really looking at these MOOCs, I would say primarily 
with the question: can you teach online courses at what we call a Penn State quality experience, 
and have them reach fourteen thousand, or fifteen thousand, or forty thousand students? That's 
a real challenge. (Ragan, Penn State World Campus). 

Evidence of improving access through MOOCs 

Having established that improving access to education and reaching a wider audience was the most 
important goal across our sample of institutions, we review here the evidence that this goal is being 
achieved. It is clear that each of the major MOOC platforms has attracted a substantial number of 
participants: around two million unique users for edX as of March 2014, and over seven million for 
Coursera. In January 2014, Coursera reported over 22 million enrollments, with many participants 
enrolling in multiple courses. These enrollees were spread across 571 courses and were based in 190 
different countries (see https://www.coursera.org/about/community) . 

While these numbers are impressive, only a small fraction of those who register complete or even start 
a course. Ho et al. (2014), using data from 17 HarvardX and MITx MOOCs offered between Fall 2012 and 
Summer 2013, found that 841,687 people registered for the MOOCs but 35% of them never engaged 
with the content, and 56% engaged with less than half of the content. Around 43,000 or 5% of 
participants completed their courses. A typical registrant was "male with a bachelor's degree who is 26 
or older" (p.2). This finding is in line with Christensen et al.' s (2013) claim that 79% of MOOC 
participants have a bachelor's degree, based on a survey of participants in 24 MOOCs offered by the 
University of Pennsylvania on the Coursera platform. 

With respect to global reach, 72% of the registrants in the HarvardX and MITx MOOCs were from 
outside the U.S. and 2.7% were from countries on the United Nations' list of Least Developed Countries 
(Ho et al., 2014). Among the 34,779 participants in the University of Pennsylvania sample, 31% were 
from non-U. S., OECD countries; 15% were from BRICS (Brazil, Russia, India, China, and South Africa); and 
20% were from other developing countries. Foreign participation is rising as MOOCs are offered in 
languages other than English. For example, edX offers courses in French, Hindi, and Mandarin, and 
recently announced a partnership with the Queen Rania Foundation in Jordan to offer Arabic language 

Hollands & Tirthali: MOOCs: Expectations and Reality 
May 2014 

57 


MOOCs via the Edraak MOOC portal. Earlier this year, Coursera announced a partnership with Fundacion 
Carlos Slim to offer courses in Spanish. 

The data suggest that MOOCs are providing educational opportunities to many individuals across the 
world. However, most are already well-educated and only a small fraction of these participants fully 
engages with the courses. Among our interviewees, we found mixed views as to whether MOOCs have 
succeeded in improving access to their institutions' offerings and reaching wider audiences. Some had 
no doubt that MOOCs had already expanded access to their courses: 

Using the technology in their courses has brought real benefits, both in terms of the effectiveness 
of the material and the reach of the course, the number of students they can serve without 
watering things down. (Fox, University of California, Berkeley). 

I will argue that we are now reaching hundreds of thousands of students a year, when previously 
we could only touch thousands of students. Also, for our subset of the education world, STEM, I 
personally argue that MOOCs are a valuable platform to extend our reach globally. Due to 
funding and other restrictions, we can only accept 10 percent international students into our 
undergrad program; now we have a way to reach those students. (Development Officer, Highly 
Selective University). 

Others acknowledged that the initial expectations that MOOCs would “democratize" education had not 
been realized: 

The idea of MOOCs is that we are going to educate the children in China and Africa. A surprising 
number of people in the Financial Engineering and Risk Management MOOC were people who 
work on Wall Street. A number of Ph.D.s were taking it too. (Sreenivasan, formerly at Columbia 
University). 

The argument for MOOCs solving the access problems of developing countries is that it is a 
solution looking for a problem, rather than the other way round: identifying what the best 
solution would be on an analysis of local conditions. It's trying to fit a sguare peg into a round 
hole. (Bates, Contact North). 

We know that there is still something about MOOCs that isn "t particularly helpful, or isn 't 
particularly appealing, to people who have not gotten a degree. (Vignare, University of Maryland 
University College). 

Quantifying the impact of MOOCs offered by a particular institution on improving access to its courses is 
difficult unless institutions accurately document the level and diversity of participation in their programs 
both before and after offering MOOCs. For the most part, institutions were not yet making a concerted 
effort to document these potential changes. One exception was a large Midwestern university where an 
evaluator described the thinking about how to capture the effects of MOOCs on the University's goal of 
expanding its reach: 

Our criterion for the second round is that the course must address and ideally solve a problem 
that would be relevant to people off-campus. We are taking a more targeted approach in what 
audience we are trying to reach, for example, people from the armed forces or college dropouts, 
and then this allows for a benchmark of success: whether we are able to attract that audience. 

Another goal is to increase enrollments. We draw well from certain states and high 
schools, but poorly from other areas and groups, so if we can start an interaction with those 
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people and understand their interests and how we could engage them, we think it will pay off 
longer term in revenues from new programs we create for them. There may be delayed benefits 
if they sign up for a program later so we would need to track this by asking what motivated them 
to sign up and whether they had participated in a MOOC we offered. (Evaluator, Large 
Midwestern University). 

At Georgia Tech, David White, Assistant Dean for Academic Programs and Executive Director of the 
Online M.S. in Computer Science (OMSCS), College of Computing, provided a solid example of how 
MOOCs can expand the university's audience. He reported a significant difference in the types of 
applicants to the MOOC-style OMSCS program being developed in conjunction with Udacity, as 
compared with the usual applicants to Georgia Tech. While typical on-campus students do not work full 
time, 75% of the 2,359 OMSCS applicants for Spring 2014 were full-time working professionals, mostly in 
computer or Information Technology fields. White also reports a difference in geographical origin of the 
OMSCS applicants: 

More than 80% are US. citizens and permanent residents compared with 75% international 
students who apply for the on-campus program. Most US. applicants are probably people who 
cannot afford the regular program or cannot uproot from a job and family to move to Atlanta. 
We think that international students may prefer to apply on-campus because they are often 
looking to work in the US. and this allows them to get a student visa and do summer internships. 
(White, Georgia Tech). 

Stan Silverman, Professor and Director, Technology Based Learning, New York Institute of Technology, 
described success in using targeted MOOCs to improve teachers' access to and use of specific 
technology tools. For example, he offered a short course for educators associated with New York State 
Teacher Centers on how to use Blackboard Collaborate as a pedagogical tool. Subsequently, he was able 
to monitor the number of requests for use of the tool and found that, compared with the usual number 
of requests, the post-MOOC uptake rate increased significantly, with more teachers using the tool in 
their professional work. 

Barriers to improving access through MOOCs 

There are many reasons why MOOCs have not fully met expectations with respect to improving access 
to education. According to Grossman (2013), most lay people outside of higher education are not yet 
familiar with MOOCs. Several interviewees pointed out that simply providing "free" educational 
resources does not necessarily make them accessible to everyone. 

The assumption of MOOCs is that we're now bringing content to the world for free, and 
everybody has the same access, so therefore we're giving everybody the same access to this high 
guality content. Yet we know from some of the work we've done in New York State and some 
other studies from around the world that the other side of the eguation isn't the same. Not 
everybody has access to the Internet or devices to play on the Internet. Who's going to pay for 
that in this model, and make sure that everyone has access to this "information highway?" (Kolb, 
RPI). 

Dr. Tony Bates, Research Associate at Contact North, Ontario, observes that the end user in developing 
countries may experience both infrastructure constraints and cultural barriers to engaging in MOOCs. In 
many locations, Internet bandwidth is inadequate for MOOC participation which involves downloading 
materials and watching videos. Buying additional bandwidth is very expensive in countries like China and 
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may be prohibitive for would-be MOOC participants. Dr. Li Fengliang, Associate Professor, Institute of 
Education, Tsinghua University, China, observed that even his own university's Internet access is fairly 
expensive and not very fast, which makes downloading materials and watching videos inconvenient. 
Adaptations have been made in certain countries to overcome bandwidth limitations. For example, 
according to Saad Rizvi, Senior Vice President, Efficacy, at Pearson, in Pakistan where the infrastructure 
outside of cities cannot support MOOCs, the government is partnering with FutureLearn and Coursera to 
transfer MOOC content to CDs. Rizvi also reported that ZayaLabs, a technology startup, provides tablets 
and a router loaded with MOOC content in a backpack to schools in India, Indonesia, and Mongolia. The 
set-up takes the content offline and allows participants to engage with it in a classroom in the absence 
of any wireless infrastructure. As platform providers such as NextThought and Coursera move towards 
mobile access to their content, some of the infrastructure constraints may be alleviated. 

Less easily addressed were some pessimistic views about the capacity of online courses, massive ones in 
particular, to reach any but the most dedicated students. Li Fengliang at Tsinghua University pointed out 
the most fundamental obstacle to MOOC participation: "/ don't think it's easy for people to learn face- 
to-face, let alone through a MOOC." Renick, at Georgia State University, observed that access to 
educational resources is only one part of the equation to guarantee success for the majority of college 
students, with the need to motivate students, track their progress, and guide them along a trajectory 
towards completion being equally important. 

Christopher Newfield, Professor of Literature and American Studies at the University of California, Santa 
Barbara, suggested that education is more complex than simply providing access to content: 

If you can get to an Internet cafe in your village, in a low-income or medium-income country... 
get access to interesting people, who are now on TV, or on YouTube. ...Going from that, to the 
interactive, cognitive, developing process, structured knowledge, and highly evolved and 
analytical processes, habits of thinking and habits of analysis that are robust and complicated 
and enable originality, that's a whole other thing. Anybody who's been teaching for a while 
knows that you can 't just put it on the screen and get that. If that were true, then we'd be a 
nation of geniuses and all our problems would be solved, all our social problems as well as our 
scientific ones. (Newfield, University of California, Santa Barbara). 

Paul Sechrist, President of Oklahoma City Community College, similarly observed that there are limits to 
what can be taught online: "For example, the biotechnology research assistant program where they are 
teaching students how to work in a sophisticated biotechnology facility: you have to have a laboratory to 
practice those skills." 

Regulatory barriers to using MOOCs for improving access to education 

In addition to logistical and pedagogical barriers to improving access to education through MOOCs, a 
number of regulatory issues also pose obstacles. For example, community colleges and state universities 
must carefully negotiate around regulatory and administrative issues related to access and state 
funding. Dr. Douglas Hersh, formerly Dean of Educational Programs at Santa Barbara City College, CA, 
noted the need for his own and other colleges to comply with the requirements of two accreditation 
agencies: Western Association of Schools and Colleges, and the Accrediting Commission for Community 
and Junior Colleges. In particular, these agencies demand that all students, whether on-campus or 
online, receive "equivalent student services" and "regular effective contact," both of which are hard to 
satisfy with MOOCs. Hersh pointed out that it would be prohibitively expensive to provide student 
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services including academic counseling, library services, tutoring, and proctoring to thousands of MOOC 
participants. 

Beth Smith, President of the Academic Senate for California Community Colleges, raised other 
"unresolved issues" such as the fact that community colleges collect apportionment or state support 
based on student enrollments in their classes. As MOOCs are offered for free, it is unclear how this will 
affect the revenue received from the state. Will colleges have to differentiate between the on-campus 
students for whom the state is paying and the out-of-state or international students? Furthermore, 
Smith points out that the faculty member unions will need to agree on contractual issues regarding class 
size, teacher workload, and expectations for assistance from teaching assistants. Laura Kalbaugh, Dean 
at Wake Technical Community College, Raleigh, NC, also notes that while it might be desirable to allow 
students to earn credits for MOOCs, articulation agreements with universities would need to be re- 
written to ensure that students could transfer these credits to the four-year colleges. 

Additional issues will arise if MOOCs become part of the K-12 education world. Lawrence Paska, Director 
of Social Studies, Harrison Central School District, Harrison, NY, and a former State Education Agency 
Technology Policy Coordinator, observes that: "In K-12, you have to remember that local taxes pay for 
local educators. The problem for K-12 will not be about the technology, but the barriers will be [related 
to] how adults have organized education." Paska raises several questions that will need to be resolved 
including: 

• How will cross-state teacher certifications be managed? 

• Who will the teachers work for and how will they be paid? 

• Will teachers become "for hire" or independent consultants? 

• If so, who would evaluate an independent consultant? 

• What will be the teacher's rights and responsibilities? 

• Who will pay for the course content and for the enrollees? 

• Will federal and state aid be available for these courses? 

• How will a teacher's course-load change from the current 150 students? 

• Will MOOCs allow for credits and, if so, who is certifying the credit and how does this work 
across districts and states? 

Regulatory issues also arise in offering MOOCs internationally. For example, in order to comply with U.S. 
Department of State economic and trade sanctions regulations, Coursera and Udacity recently restricted 
access to their courses in Cuba, Iran, and Sudan (see Curley, 2014 ). It appears that despite a declared 
commitment to providing global access to free education, many obstacles remain in fulfilling this goal. 

Victor Vuchic, formerly Program Officer for Education at the William and Flora Hewlett Foundation, who 
questions Silicon Valley's "blind faith that technology helps everybody," offered a verdict on how well 
MOOCs improve access to education: 

I read this article a year ago when the first MOOCs came out; a journalist said "My 9th grade son 
took two Stanford MOOCs and completed them." I thought, "Right now, every wealthy parent 
just hired tutors for their kids to sit and complete four Stanford MOOCs to put on their college 
application. The bar at every elite school just went up and none of the poor kids have any chance 
to do that..." It is true that technology is going to give people that have nothing, something. 
People across India now can have Wikipedia and CDs of Sal Khan videos. Technology raised the 
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floor - but it is going to blow the roof off the top. (Vuchic, formerly at William and Flora Hewlett 
Foundation). 

We concur with Vuchic that, overall, the evidence suggests that MOOCs are currently doing more to 
increase gaps in access to education than to diminish them. 
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Case 1: 

Virology for the Masses: 

Vincent Racaniello, Columbia University 

Virology I: How Viruses Work was among the first five MOOCs offered by Columbia University and one of 
the first in a series of two MOOCs offered by Professor Vincent Racaniello, based on the video 
recordings of his on-campus course Virology. The MOOC Virology I: How Viruses Work was offered over 
11 weeks starting in August 2013, and the sequel, How Viruses Cause Diseases, was offered over 12 
weeks starting in January 2014. In addition to the video recordings, the MOOCs draw on Professor 
Racaniello's virology blog (virology.ws) and his weekly podcast This Week in Virology. Professor 
Racaniello describes his motivation for using technology to reach the masses as follows: 

“My goal is to teach as many people as possible about viruses, which is unusual for a researcher. I was 
teaching and I realized that I can explain complicated things quite easily. That is a gift for sure. You have 
to use it. Same with the knowledge. I find no reason to keep it to myself. If I can teach 180 students a 
year but can reach thousands going beyond the classroom, then why not?" 

I have been a basic researcher my whole career here but I always liked teaching. I gave lectures all over 
New York: Cornell and Rockefeller and so on. A few years back I was involved in writing a textbook on 
Virology. Having written that, I got a very broad view of virology. That is the background. Then blogging 
came to be available for the masses. In 2004, 1 started blogging - might as well share some knowledge of 
virology with the world, I thought. I started writing just in a general way that goes to inform the public 
about viruses. I developed a following on that blog over the years. Today, I get two or three thousand 
hits a day. Not bad for a highly technical subject. 

Then I started to podcast. I thought that it was another medium I could use so I got all this audio 
equipment. I do three different podcasts every few weeks. My colleagues and I have conversations about 
topics in virology, parasitology, or microbiology. They are called "This Week in 
Virology/Parasitism/Microbiology." It is a totally different audience than blogs. Some people love to 
read, other people like to listen. The virology podcasts, for example, get 80,000 - 100,000 downloads a 
month so we have a big audience out there. 

Then four years ago I decided. ..Columbia does not have a Virology course. It is one of the best institutions 
in the world and does not have a virology course. I submitted a proposal to do one and it was accepted. 
The first year I had 30 students. The following three years I had 60, 90, and 180 students. From the 
beginning I decided to record the lectures. Just for the students so that they are able to listen to it. It is a 
complicated subject and I don't expect them to get it all in the lecture. So I thought of recording them 
and posting them. I started posting them on iTunes U. 

A lot of people who have seen the iTunes U lectures said, "You should do a Coursera course now and 
reach more people." So I viewed it as an expansion of iTunes in a way. Although, in two years I got 
100,000 enrollments on iTunes and Coursera has 26,000 students enrolled, so it is less than iTunes. But in 
iTunes it is hard to know how people are using it. You don't get feedback. With Coursera, you can see 
how many people are taking the quizzes. There is a discussion board so you can sense who is engaged. 
Part of the reason why people are engaged is because they rarely had access to a professor outside the 
college. They [professors] are too busy to bother with e-mails and engaging with the public. I am really 
happy to do that. I think it is important. 
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I don't do Coursera for Columbia students. However, my ability to do Coursera is based on the Columbia 
course. I develop the course and use the lectures from it. I stay in front of my class and lecture and I am 
inspired by seeing them [the students]. It makes me a better lecturer. Many people record their lectures 
for Coursera with nice lighting and camera work. I think that it is flat. The best part is that the 
participants can see that I am in front of the class; they can hear students asking guestions. For me it 
gets me excited. 

I think complicated subjects like science are best taught by people who are doing the science. Certainly, 
other people can teach virology, but they don't bring the same depth of experience and passion. I think 
that is what the students feel and the public feels: the passion. You cannot have it if you just learned 
from a textbook. When possible, scientists doing the research should be doing the teaching. That is my 
mantra: to bring what I do in the lab to the public. They like the insight from the lab. They pay for the 
research so they need to know what is going on. They can't read it in the journals unfortunately as it is 
behind a pay-wall. So that is what we do. We discuss the literature so that people have access to it. 

My goal is to be the earth's virology professor. One of my Spanish colleagues said "You can't be the 
earth's virology professor because you just talk in English." So I said, "You can translate it if you like." So 
she got her students to translate it into Spanish. It would be great if it could be translated into Russian, 
Chinese, and Portuguese also. Italian I should be able to do myself. 

My goal is to take this course I develop here and give it to the world. I think everyone has a right to 
knowledge. The more people in the world we make smart, the better workforce we have. It is very clear 
that knowledgeable people will contribute to the better health of the world. Even if they can ’t afford it 
[education], they should have access to it. Giving it to them is just going to enrich the world. In my case, 
in the subject of virology, it is very relevant as everyday people in the world are faced with issues of 
virology. For example, Polio virus has been isolated in Israel in the last few months. I have been getting e- 
mails asking what this means. What do we have to do? That is the kind of real-life stuff I try and teach 
people. 

Every aspect of virology is important, not just how we get sick and preventing it, but also how so much of 
our genome is virus and what does that mean. I end up turning people on to the subject. People say I 
want to be a virologist because of your course. If I can inspire one person to be a virologist, that is the 
best thing I can do. 

Source: Vincent Racaniello, Professor, Columbia University. 
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Case 2: 

Developmental Education MOOCs: 

Cuyahoga Community College's Pre-Algebra CCC 

Cuyahoga Community College, OH, (Tri-C) received a grant from the Bill & Melinda Gates Foundation in 
November 2012 to create and facilitate a developmental education math MOOC, Pre-Algebra CCC. The 
MOOC was targeted at students performing at the lowest level of developmental education in math. 
Students placed at this level need three semesters of developmental education courses before they can 
enroll in college-level math courses. The goal of offering the MOOC was to leapfrog students out of at 
least one level of developmental education, reducing the delay in earning credits towards a degree. 

For community colleges, helping underprepared students through the hurdle of developmental 
education courses is an important aspect of improving access to college education and ensuring 
successful completion of a degree program. According to Kalbaugh, Dean, Wake Technical Community 
College, Raleigh, NC, if students become mired in developmental education courses, they may not reach 
the level of credit-bearing courses at all or may not persist through to completion because they have 
already expended substantial time and resources in college. Therefore, helping students advance to 
credit-bearing courses sooner is the key to keeping them on track for successful completion of their 
degree program. Belinda Miles, Provost and Executive Vice President of Access, Learning, & Success, 
Cuyahoga Community College, OH, points out that the number of students who must repeat a 
developmental course has been very high at Tri-C. As an "Achieving the Dream" college, Tri-C has been 
working on finding ways to improve student outcomes and shorten their time in developmental 
sequences. 

Tri-C's Pre-Algebra CCC MOOC was hosted on CourseSites, the open platform offered by Blackboard. The 
four-week long MOOC was offered in March, April, May, and June 2013. The MOOC was different from 
most xMOOCs in two ways. Firstly, it used only pre-existing open educational resources, including videos 
from Khan Academy and TeacherTube, freely available exercises, and OpenStudy to facilitate 
collaboration among students. Secondly, the entire MOOC was structured as a game with a wrap- 
around story. Custom graphics were donated by an artist. The game mechanics included leveling-up, 
that is, students could not get access a higher level until they had completed the previous one. The bar 
for leveling-up was set high at 80 percent. 

According to A. Sasha Thackaberry, District Director, eLearning Technologies, Cuyahoga Community 
College (Tri-C), OH, gamers in general fail 80% of the time and succeed the other 20%, and yet they 
develop a remarkable persistence, resilience, and sense of agency - the very skills community colleges 
need to foster in their students. Miles sees gamified learning experiences as a "safe-failure 
environment." Students might not feel comfortable failing 80% of the time in front of their peers and 
teachers, but when they are working independently and trying to beat a machine or game, they are 
motivated. Typically, students who are working on this type of college-readiness course experience a 
great deal of math anxiety. The Tri-C instructional designers believed that the game environment would 
be a good choice for this population. 

After the first four facilitated iterations of Pre-Algebra CCC, the course has been kept open on 
CourseSites (https://tric.coursesites.com/), but without facilitation from Tri-C faculty. Tri-C is collecting 
data to ascertain student success in the unfacilitated version. 
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A total of 1,372 students signed up for Pre-Algebra CCC during the four facilitated iterations, with 
enrollment rising from 138 in the March offering to 537 in the June offering. Students were asked to 
complete a survey before being given access to the course materials. The survey queried the students' 
demographics, background experiences, and reasons for taking the course. Eight hundred and nine of 
the enrolled students completed the survey and were able to participate in the course. Eleven percent 
of the initial enrollees completed the course and among those who actually participated, 18% 
completed with a score of 80% or above. Tri-C set the passing bar higher than the typical 70% for 
passing math courses because of historical Tri-C data indicating that students who earn only a "C" grade 
in the first-level developmental education course are far less likely to be successful at the next level. Tri- 
C wanted to ensure that students completing Pre-Algebra CCC would also be successful in future 
developmental courses. 

Among the 147 Pre-Algebra CCC completers, 13 took a math placement test before the MOOC as well as 
after. Of these, seven performed well enough on the second placement test to skip one developmental 
education math course and three were able to skip all three required developmental education courses 
and move directly to credit-bearing courses. While the numbers are small, they indicate that the goal of 
reducing the number of full-length, developmental education courses students need to complete was 
certainly feasible, saving students both time and money. 

Miles points out that the participants in developmental education MOOCs represent a totally different 
population from those participating in MOOCs created by universities. The latter, often strongly- 
branded institutions, attract many college graduates who, according to Thackaberry, "live in a highly 
networked environment themselves" and "find out about MOOCs with relative ease." In contrast, 70% of 
those Pre-Algebra CCC participants who recorded their highest level of education in the pre-survey had 
not yet earned any degree and, most likely, have relatively few sources of information regarding 
educational opportunities. Tri-C is determined to provide access to "self-remediation opportunities" for 
those who are not already connected to "informal academic learning environments." 


Sources: 

Belinda Miles , Provost and Executive Vice President of Access, Learning, & Success, Cuyahoga Community College, 
OH; 

A. Sasha Thackaberry, District Director, eLearning Technologies, Cuyahoga Community College, OH; 

Laura Kalbaugh, Dean, Wake Technical Community College, Raleigh, NC. 
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Goal 2: Building and Maintaining Brand 


For institutions of higher education, building and protecting brand serves to attract and retain students, 
faculty members, and partnership opportunities with other institutions, funders, and alumni networks. 
Interviewees from 41% of the institutions in our study sample that were offering or using MOOCs stated 
that branding, positioning, or attracting students was a strategic goal for the initiative. These 
interviewees represented seven public universities, three private universities, and a museum. None of 
the community colleges mentioned branding as a goal for engaging in MOOCs. 

Branding implies different strategies to different types of institutions, depending on their current 
positioning. For flagship state universities in the U.S. that have already established local appeal, MOOCs 
allow for more national and international recognition. For elite institutions, being at the forefront of a 
highly publicized innovation serves as a signaling mechanism to protect their global ranking among the 
top universities. Additionally, the act of offering free education to the world helps to soften their 
exclusivity and wealth, and justify often disproportionate research funding received from public entities. 
Community colleges are less concerned about branding than improving access to education for an 
underprepared student population, but Judy Baker at Foothill College notes that offering MOOCs could 
help “dispel myths about community colleges and help people understand what community college 
means." For European universities which must generally educate domestic students for free or at very 
low tuition rates, MOOCs may help attract international students who are an important source of 
income. 

Institutions that were employing MOOCs as a vehicle to expand their brand made strategic decisions to 
showcase programs, specialties, and research capabilities in which they believed they were leaders or 
wanted to be among the first to stake a claim to world-class expertise. For example, Ragan at Penn State 
World Campus suggested that offering the MOOC Epidemics - the Dynamics of Infectious Diseases was 
"a way to carve out that space" and make "a claim to a domain." Administrators often call on "star" 
faculty members to offer the institution's first MOOCs. Martin at West Virginia University described a 
cMOOC on the topic of social media that was hosted by multiple faculty members in the Department of 
Communication Studies as a means to promote the department and improve awareness of what it does 
to the general public. Deborah Keyek-Franssen, Associate Vice President for Digital Education and 
Engagement, University of Colorado System, used the term "evangelist courses" to describe the MOOCs 
being offered to promote the “awesome things we are doing and want everyone to know about." Lester 
at FutureLearn similarly believes that MOOCs provide a useful vehicle for "broadcasting" cutting edge 
research to experts around the world, both enhancing the originator's reputation and serving to attract 
collaborators to help extend the work. 

According to Nick Hathaway, Vice President for Administration and Finance at the University of 
Oklahoma, if an institution claims to offer a premier program, failure to offer an online version in the 
near future will incur reputational risk, especially if another university offers an online course in that 
domainand "champions themselves as the center of the universe for that subject." Sir John Daniel, and 
Lester at FutureLearn made related observations regarding the need to protect brand by staying at the 
forefront of new trends. 

The Open University is doing this because it's kind of like MOOCs are the territory of what the OU 
is all about. ..It would be a lost moment if the OU didn't participate in MOOCs. It's kind of what it 
does. (Lester, FutureLearn). 
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Reputations are at stake, especially for the lesser institutions. The situation at the University of 
Virginia is a case in point where the President, Teresa Sullivan, was fired and then quickly re- 
hired, with concerns cited by the Board that included not moving quickly enough into online 
education. (Daniel). 

The one museum we invited to participate in this study, the American Museum of Natural History 
(AMNH), NY, emphasized branding as its primary goal for offering MOOCs, with the specific aim of 
highlighting its capacity to offer high quality, structured, educational experiences led by instructors who 
are experts in their field (see Case 3 ). 

...obviously we care what people learn but we also deeply care about their perception of the 
museum as a valuable entity in their lives. ..How is this helping the museum count more in their 
assessment of their learning continuum?. ..Most people in our course didn't know that we existed 
before, or that our resources existed. ...That's the main benefit for the museum. (Kinzler, AMNH; 
Steiner, AMNH). 

Some interviewees offered less flattering perspectives on the branding potential of MOOCs, with a few 
suggesting that most institutions pursuing them were “being sheep" and others accusing some MOOC 
instructors of being primarily interested in being “recognized in the parking lot," “defending their 
textbooks against interlopers," or expanding the market for their textbooks. Bates suggests that "Ivy 
League universities are trying to use MOOCs in such a way that they now 'own' online learning in ways 
that enable them to leverage their brand." At worst, MOOC initiatives were viewed as “a demonstration 
of offensive, neo-imperialist hubris. "( Daniel). 

MOOCs as tools of recruitment 

Building brand is a direct pathway to increasing recruitment and enrollment of students in tuition- 
earning courses and programs. MOOCs offer participants a "frictionless," that is, low-risk, no-cost 
introduction to a program area, and the offering institution can simultaneously identify candidates who 
might perform well as degree-earning students. For example, at the University of Colorado Boulder, 
College of Engineering professors who offered engineering courses as MOOCs were planning targeted 
recruitment efforts that would involve sending information packets about the program to top 
performers in the MOOC. Similarly, one of the goals for the Geospatial Analysis and Mapping MOOC 
offered by Penn State World Campus was to increase awareness and interest in the area to lead people 
to the Master's Degree in Geoscience Studies. Ragan at Penn State World Campus mentioned that, after 
the MOOC ran in July 2013, there was a substantial increase in inquiries and interest in this program, 
with traffic on the program website tripling. 

An evaluator at a large Midwestern university is analyzing participation in the university's MOOCs and 
gathering survey feedback from MOOC learners regarding their potential interest in other courses and 
programs, with the aim of creating attractive offerings that will lead to increased enrollments at the 
university from outside the state. “An extensive end-of-course survey posed many psychoqraphic 
questions, many of them open-ended. We got responses indicating that participants wanted more 
sequenced courses.” Similarly, Ragan at Penn State World Campus noted that course surveys are being 
used to elicit ideas for future offerings that would be appealing to MOOC participants. 

Academic Partnerships, a for-profit, online service provider and technology "enabler," has formalized 
the route from MOOC to degree with its MOOC2Degree initiative which offers an initial course from a 
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partner university's online program free to non-enrolled participants (see Case 4 ). If successful in the 
course, these participants can earn credit for the MOOC by enrolling in the full program at the offering 
university. Currently, two state universities are offering MOOCs through this initiative. The College of 
Nursing at the University of Texas at Arlington has piloted an offering of its course Enhancing Patient 
Safety through Interprofessional Collaborative Practice through MOOC2Degree. Successful completers 
may apply to the college's Registered Nurse to Bachelor of Science in Nursing (RN to BSN) online 
program. The University of Cincinnati is offering Innovation & Design Thinking which can lead to 
entrance into the Master in Business Administration or Master of Engineering online degrees. 

Hersh, formerly at Santa Barbara City College, views MOOCs as "virtual land grabs," that is, 
"opportunities for universities to brand themselves to the rest of the world to increase their student base 
and to develop additive revenue." He notes that, prior to the advent of distance education, many 
universities attempted physical expansions abroad, but that with "distributed management" and foreign 
cultures, these satellites were often unsuccessful. Hersh suggests that MOOCs may provide a more 
manageable alternative. 

In addition to recruiting students, some institutions emphasized the need to showcase themselves as 
the leader in academic content areas and in pedagogical innovation so as to attract and retain faculty 
members who expect "to play in the same league as their competitors." (Mintz, University of Texas 
System). MOOCs also offer an opportunity to maintain connections to alumni, who are an important 
source of funding. 

Several interviewees commented that MOOCs provide opportunities to forge mutually beneficial 
partnerships with other educational institutions - what Lester at FutureLearn described as "coopetition." 
They also facilitate relationships with corporate partners and funding agencies, potentially leading to 
increased grant revenues. For example, the business school at the University of Colorado Denver, 
offered the Global Energy Management (GEM) MOOC in order to create brand recognition for the 
School of Business in the energy field and to reach out to energy professionals in the region who could 
offer potential for collaboration or funding. Mintz at the University of Texas System believes that the 
university's investment in edX has been worthwhile in establishing it as a serious player in the realm of 
educational technology and innovation: 

It has made us competitive for national grants that we would not have been competitive for 
otherwise. It put us in conversations that we wouldn't have been in. We are working closely with 
MIT, Berkeley, Harvard, Georgetown, Rice, and others in ways we would not have been working. 
(Mintz, University of Texas System). 

Some institutions, recognizing that existing brand is a key factor in the success of MOOCs, are saving 
their dollars for other initiatives where they have more competitive edge: 

I am a realist about the branding issue... it would be very difficult for a MOOC from Georgia State 
to become among the definitive MOOCs in some major field like introduction to psychology when 
there is Duke and Harvard and Stanford and Michigan and others who are producing their own 
versions of this course. This is not to denigrate Georgia State, I am very proud of the institution 
and the commitment we make to our students, and the success they've had. But it's being 
realistic and if we have $2mm, we can put it in programs that will change the lives of hundreds 
or thousands of students, or we can develop one or two MOOCs that likely will soon be 
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overshadowed in the marketplace by the production of better branded institutions. (Renick, 
Georgia State University). 

Isolating and measuring impact of a new initiative on brand can be notoriously difficult. An evaluator 
from a large Midwestern university suggested the possibility of capturing the effect of offering MOOCs 
on brand and reputation by comparing historical data on applications and admissions: "We could look at 
how many minorities get involved, how many people from other countries, from our state and other 
states. We can look at whether applications and admissions go up or down overall and for certain 
groups." While we heard substantial anecdotal evidence and many aspirations with regard to the 
positive impact of MOOCs on brand most institutions are only just beginning to think about how to 
capture and quantify branding-related benefits to determine whether they outweigh the costs of 
investing in MOOCs. 

As institutions think about extending their brand through MOOCs, one complication that arises is the 
question of where the brand actually lies. Do participants opt for a course because it is on a particular 
platform or because it is offered by a particular university? As MOOC platform providers such as edX and 
Coursera open their platforms to a wider range of institutions, some of the initial cachet of belonging to 
these consortia is being lost. A senior administrator at a leading university in the United Kingdom 
expressed strong reservations about partnering with one of the MOOC providers for fear that it might 
dilute the institution's own very strong brand. 

We would be hesitant about getting involved with any of the current big providers of MOOCs 
because we think the governance challenges involved in associations with edX or FutureLearn in 
the UK are considerable for a University like ours which has a very strong brand identity. We 
don't dilute or give out that brand identity lightly so there would be a lot of bridges for us to 
cross to be associated with the current providers. We are not going to be railroaded politically 
into getting into MOOCs. (Senior Administrator, Leading UK University). 

A final challenge in using MOOCs to build and maintain brand is the clear tension that Chris Dede, Wirth 
Professor in Learning Technology at Harvard University, observes between building brand and increasing 
access: "The problem the MOOCs are underscoring is that Flarvard has contradictory goals of increasing 
its footprint and protecting its brand. Brand has value by being limited. The two goals paralyze each 
other." 
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Case 3: MOOCs to Increase Visibility: 

American Museum of Natural History MOOC Initiative 

In early 2013, Coursera invited the American Museum of Natural History (AMNH) in New York City to 
create three science-oriented MOOCs to be delivered in the Fall of 2013 through Coursera's Teacher 
Professional Development program. AMNH successfully launched three 4-week long MOOCs targeted at 
science educators: Genetics and Society: A Course for Educators, The Dynamic Earth: A Course for 
Educators, and Evolution: A Course for Educators. While the museum had several goals for this initiative 
including improving teachers' science content knowledge and use of AMNH materials in their 
classrooms, their primary goal was to increase the museum's visibility, especially with science educators 
and international audiences. Some indicators of success in achieving this goal include the following: 

• At time of offering the three MOOCs, AMNH was one of only three museums worldwide 
represented in the Coursera Teacher Professional Development program, accounting for three 
of the 44 MOOCs available. 

• A total of 39,685 people registered for the three courses. 

• Around two-thirds of these registrants were international, representing 110 countries. This 
contrasts significantly with the almost purely domestic population of participants in the 
museum's regular online courses. 

• Sixty-two percent of the survey respondents had no prior experience with the museum or its 
content and were therefore "new customers." 

• Forty-two percent of the MOOC participants who responded to surveys at the start of each 
course were educators, the target population for the AMNH MOOCs. 

• While the vast majority of survey respondents learned about the AMNH MOOCs from the 
Coursera website, some heard about them from a friend or colleague, an online social network, 
a search engine, blog or press article, AMNH's own website or from their school or school 
district. 

• The promotional videos for each of the three MOOCs were discoverable on YouTube outside of 
the Coursera platform and were viewed between 450 and 864 times. 

Sources: 

Dr. Ro Kinzier, Senior Director, Science Education, American Museum of Natural History, NY; 

Dr. Robert Steiner, Director, Online Teacher Education Programs, American Museum of Natural History, NY; 

Maria Janelli, Senior Manager, Online Teacher Education Programs, American Museum of Natural History, NY. 
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Case 4: 

MOOCs for Recruitment: 

Academic Partnerships MOOC2 Degree Initiative 

One of the most strategic approaches to recruiting new students into university programs through 
MOOCs is being led by Academic Partnerships (AP), a for-profit entity that works primarily with public 
universities in the U.S. and universities abroad to: 

• Assist faculty members with conversion of their traditional degree programs and courses into an 
online format. 

• Recruit qualified students into these programs. 

• Help with retention of students through to graduation. 

• Expand the reach of partner universities globally through an established worldwide university 
network. 

AP's mission is to utilize technology to expand access to high quality, post-secondary education at scale. 
AP's Executive Vice President, Dr. Justyna Dymerska, believes that MOOCs provide the capability to 
disseminate knowledge at a very low cost. However, she cautions that the current model of providing 
MOOCs for free is not sustainable for public universities that are already burdened with increasing costs 
and shrinking funding. To "capitalize on the potential of MOOCs while addressing the bottom line," AP 
launched the concept of MOOC2Degree in January 2013. 

Under this initiative, the initial core course in an online degree program that AP represents is converted 
into a MOOC. These MOOCs are free, open, and "full-calorie courses," that is, they have the same 
academic content and instructor as the closed, credit-bearing, online course. Students who successfully 
complete a MOOC2Degree course can apply to the full degree program to which the course belongs, 
and, if admitted, receive credit for the MOOC based upon criteria established by the partner university. 

According to Dymerska: 

A MOOC is an example of an innovative disruption within the higher education model. It 
increases the branding and makes our partner institutions more competitive in a very dense, 
competitive higher education market. The partner universities see it as a practical approach, not 
only for increasing local market share, but for global growth as well. It fits the mission of 
expanding access. ...From a more operational perspective, they get a free look at a student and a 
student's performance in a course. ...Essentially it's a filter for the university. Students who carry 
on into a degree program are high performance students, they're mature and they want to 
complete a degree. 

From AP's perspective, "MOOC2degree is a great recruiting tool for students, allowing universities to 
cast a broad net. We let people take the online version of a course and those who wish to continue do so, 
and for those who don't, they've had a great learning experience." (Dymerska, Academic Partnerships). 
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To date, the following institutions have launched or planned a MOOC under the MOOC2Degree 
program: 

• University of Texas at Arlington College of Nursing created the MOOC Enhancing Patient Safety 
through Interprofessional Collaborative Practice leading to the RN to BSN (Registered Nurse to 
Bachelor of Science in Nursing) online program. 

• University of Cincinnati created the MOOC Innovation & Design Thinking leading to the Master 
in Business Administration or Master of Engineering online degrees. 

• Lamar University (Proposed). 

• University of West Florida (Proposed). 

The College of Nursing at University of Texas at Arlington (UTA) initially teamed with AP to convert their 
traditional RN to BSN program into "an innovative, highly-accessible, affordable, and scalable [online] 
program" (Poster, Mancini, & Ganji, 2013). The RN to BSN online program enabled flexible scheduling 
and competitive tuition, addressing two barriers identified for working nurses with diplomas or 
associate degrees who wish to pursue advanced degrees. 

MOOC2Degree was proposed by AP as one of the ways to recruit students to this program and the 
course Enhancing Patient Safety through Interprofessional Collaborative Practice was selected to be 
offered as a MOOC. The target audience was practicing nurses, with a view to providing them an 
opportunity to determine whether online learning works for them, to experience the effort required, 
and to make an informed decision about committing to a full degree program. The MOOC2Degree 
option lets them try out the first course without any financial commitment to a program. 

To receive credit for the MOOC, students must: 

• Complete the MOOC2Degree course with a score of 80% or higher 

• Complete an online proctored exam within seven days after the course ends with a score of 70% 
or higher 

• Enroll in the RN to BSN program after fulfilling the admissions criteria. Learners who wish to 
receive Continuing Education Units for the course pay the continuing education provider $25 to 
obtain a certificate for the 45 hours of continuing education credit. 

In August 2013, UTA's Enhancing Patient Safety through Interprofessional Collaborative Practice MOOC 
went live with a "soft launch" to pilot the course structure and processes. On the start date, 
approximately 300 learners were enrolled. While AP did not disclose how many of the enrollees were 
actually registered nurses, they reported that all of the registered nurses who responded to a 
participant survey within the MOOC expressed interest in applying to UTA College of Nursing's RN to 
BSN program and receiving academic or continuing education unit (CEU) credit. By the time the MOOC 
closed, 50% of these registered nurse survey respondents were moving forward with the application 
process. AP will receive a percentage of the revenues from tuition paid by these students. 


Sources: 

Dr. Justyna Dymerska, Executive Vice President, Academic Partnerships. 

Poster E. C., Mancini M., Ganji, D. (2013, August 5). MOOCs and more: Expanding online access for nurses. 
EDUCAUSE Review Online. Retrieved from: http://www.educause.edu/ero/article/moocs-and-more-expanding- 
online-access-nurses 
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Goal 3: Improving Economics: Reducing Costs or Increasing Revenues 


Thirty-eight percent of the institutions in our sample that were offering or using MOOCs, and twenty- 
nine percent of our interviewees overall, claimed that a goal for their MOOC initiatives was to lower 
costs or increase revenues, or both. Among the 29 institutions currently offering or using MOOCs, this 
goal was mentioned by representatives of eight of the 15 public, four-year universities, two of the ten 
private, four-year universities, and one of the three community colleges, suggesting that it was more 
important to public universities than private ones. There was widespread acknowledgement that the 
current expenditures on MOOC development could not continue indefinitely without financial 
justification. 

The States used to cover about 75% of students' tuition but now it's more like 25% in most 
states; at least that's true in Michigan where I was before. As that is happening, we [the San Jose 
State University administration] feel it is incumbent upon us to research and to figure out how 
we are going to survive as an institution. (Cheal, San Jose State University). 

We'd be crazy not to be thinking about cost-effectiveness given that state funding has dropped 
and will probably not rebound to earlier levels. (White, Georgia Tech). 

Our primary goal is to find cost-effective, scalable ways of providing professional development. 
(Kleiman, North Carolina State University). 

Online tools will fundamentally increase the value students get from residential education. I feel 
this will address many of the vexing questions about the costs of higher education. (Sarma, MIT). 

Some interviewees suggested that the notion of cost savings through MOOCs arose primarily from 
policymakers looking for a silver bullet to reduce state education outlays: 

...given the severe budget crunch that California is under and the severe crunch in higher 
education, [the governor is] convinced that online learning is part of the answer and that it's 
needed. He has been heavily promoting it. (Means, SRI). 

A lot of politicians and business leaders really thought that you can essentially just do this and 
solve universities' cost disease: do to university what has been done to lots of other industries, 
with digital technology. Getting people out of it. Getting excess labor costs out of it and get a 
better quality product. That was the claim: better education for less money. The promise was 
solving the quality-cost tradeoff. (Newfield, University of California, Santa Barbara). 

For high-level policy makers, the possibility of bending the cost curve does seem to be the 
dominant reason why they are interested. (Professor, East Coast University). 

Many interviewees expressed concern at the lack of sustainability for MOOC initiatives given the heavy 
burden on faculty time and other institutional resources. 

If MOOCs are going to demonstrate a genuine return on investment, they must produce 
educational resources that can make a genuine contribution to learning across the campuses, for 
example, by developing interactives, simulations, and other educational materials, and develop a 
next-generation platform that can support state-of-the-art teaching innovations. (Mintz, 
University of Texas System). 

Here is the conflict that we have today in higher education: cost is going up. I believe it is up 
either four or five times since the 80s. It went up by 8.3% in 2012. Government subsidies are 
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down. Just last year in 2012, they went down by about 9%. The middle class is tapped out with 
one trillion dollars in student loan debt, more than all consumer debt in the United States. This 
unfavorable landscape is accompanied by declining enrollment, which I believe last year went 
down by 2.3%. 

When we assess the MOOC concept and look at it from this perspective, if a university 
provides credit for a MOOC, then the model becomes problematic because it takes a significant 
bite from the operating revenue of a university. Giving credit for the MOOC is actually increasing 
the cost of delivery. Basic economics here, when you think about MOOCs, is that if you reduce 
the price of your product in any sector, not only in higher ed., but any industry, you have to lower 
the cost of production for it to be sustainable. In our view, universities have been defying gravity 
here by marking courses, their core product, to zero. This is unsustainable. We see the MOOC in 
its current form as a catalyst that shortens the time of sustainability of today's already broken 
university business model. Increasing access cannot be sustained by widening their deficit. We 
see MOOCs as potentially threatening to the university's mission unless they are tied to degree 
programs and are used in the context of increasing paid enrollments. (Dymerska, Academic 
Partnerships). 

Burck Smith at Straighterline categorizes MOOCs, along with other online education interventions, as 
"marginal cost-pricing models," and similarly points out a conflict between this model and the way 
universities and colleges currently manage their finances. 

What you're seeing is these entities are pricing individual courses much closer to their true cost 
of delivery. It really doesn't cost much to deliver that on a marginal-cost basis. There are some 
fixed costs for sure. But one of the central tenets ofecon 101 and microeconomics, is that the 
price of a good ultimately equals its marginal cost of delivery. As a categorical definition, all of 
these are marginal cost-pricing models. That does not fit at all with how higher education is 
constructed. If you look at colleges who are offering online courses, they are pricing them 
oftentimes ten to thirty times higher than their true cost of delivery, making big profits. That 
subsidizes the face-to-face environment, or subsidizes other parts. What the "unaccreditabie 
sector" is doing, is simply pricing the individual units very close to the cost of delivery. (Smith, 
Straighterline). 


Potential Cost Savings from MOOCs 

In spite of recognition that MOOCs have, so far, proved to be a significant drain on time and money for 
institutions, interviewees offered several possibilities for eventual cost savings to institutions: 

• Re-using MOOC materials multiple times 

• Sharing MOOC materials across instructors and campuses 

• Developing common courses to offer across institutions 

• Replacing on-campus courses with MOOCs 

• Faculty time savings 

• Reducing the need for facilities 

• Recruitment efficiencies 

• Less costly student support services provided by non-faculty members 

• Increasing student throughput 
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Each of these ideas is described in greater detail below. Not surprisingly, given the infancy of MOOC 
initiatives, we were hard pressed to identify examples of actual cost savings realized to date. On the 
contrary, we were able to gather significant evidence of the high costs of development and delivery of 
MOOCs which we present in a later section: Resource Requirements and Costs of MOOCs . 

Re-using MOOC materials multiple times 

Several interviewees argued that, after the initial investment in creating online materials, re-using them 
with limited revisions would improve cost-efficiency: 

... but over time, certainly, once the higher costs of development are over, it would certainly pay 
off if the materials don't have to be changed very often. (Cheal, San Jose State University). 

How you repurpose that MOOC afterwards has to be part of the cost-effectiveness eguation. 
(Fisher, Vanderbilt University). 

James, of Mt. San Jacinto Community College, contrasts the costs of offering a developmental writing 
course on-campus with the costs of re-running the MOOC, Crafting an Effective Writer, recently 
developed by the college: 

If this MOOC helps our students, it is going to save us money. We wouldn 't have to have these 
courses on-campus. It costs us about $6,000 every time we offer it on-campus to pay instructors 
and tutors. There is a cost to re-offering the MOOC, but not a huge one. We already created it. 
There is not going to be huge change. We paid our tech 20 hours at $45/hour to renovate the 
course for the second offering, to make revisions and keep track of things. The cost of running it 
is not that great. The cost savings are greater. (James, Mt. San Jacinto Community College). 

Ragan at Penn State World Campus, based on his experience with regular online courses, is less certain 
about being able to re-use MOOC materials without significant adjustments each time. In particular, he 
notes that designing formative and summative assessments and strategies to engage students in 
interactions and dialogues around the content are “rarely correct the first time out." 

In the best of all worlds, you design it once, develop it once, and deliver it four times or five times 
before you pull it back up under the rack and rework it. The truth of the experience has been for 
us always in online courses is, the first three or four times you teach the course, you deliver the 
course, there are significant modifications made on the revision. You run it one time, you figure 
out what didn't work. You come back, you retool it, and so forth. You're automatically into a 
revision cycle that can be as much as a half of the initial investment. It's not until about the third 
or fourth time until it begins to stabilize some. By then, your content's old. (Ragan, Penn State 
World Campus). 

Sharing MOOC materials across instructors and campuses 

Several interviewees suggested that MOOCs could be used across multiple instructors and multiple 
campuses. In some instances, the MOOC materials developed by one instructor could, with appropriate 
licensing, be embedded in a course delivered by another instructor. In other cases, one single course 
could be offered online across multiple campuses with no local instructor. MJ Bishop, Director, Center 
for Innovation and Excellence in Learning and Teaching at the University System of Maryland, observed 
that in these latter situations it would be necessary to resolve logistical issues such as how tuition and 
credits are transferred. 
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Rebecca Griffiths, Program Director for Online Learning at the research organization Ithaka S+R, which 
has conducted several notable studies of online learning, envisions cost-efficiencies in repurposing 
MOOC materials for multiple contexts. Amy Collier, Director of Digital Learning Initiatives, Stanford 
University, concurs and Guthrie, President of ITHAKA, provides a specific example of how this might 
happen. 

\Ne think it does not make sense for every instructor to create their own set of MOOCs or regular 
online materials to flip their courses or to try new formats. There should be some efficiency in 
repurposing materials created by one set of instructors and using them in a lot of courses - the 
marginal costs of using them are close to zero. Once you have made the investment in what 
should be very high guality materials, there is an argument for using them in as many contexts 
as possible. (Griffiths, Ithaka S+R). 

A lot of MOOC materials can be re-purposed, re-seguenced, reconfigured for on-campus 
students, for public audiences, for professional audiences, without too much of a cost impact. 
(Collier, Stanford University). 

Let's say that I'm a computer science professor. I already know how to teach programming, but I 
don't teach Python. I can have my students take the Python MOOC from Rice University. Then I 
can add something to that MOOC, maybe create some other assignments. That provides me with 
an opportunity potentially to offer a better learning experience at significantly less cost. 

(Guthrie, ITHAKA). 

Interviewees from some university systems see the potential for cost savings if educational materials 
can be shared across the system campuses or re-used many times over without significant changes. 

I think we're more interested in creating repositories of open educational resources and finding 
places where we can enhance our efficiency by sharing resources across the system. (Bishop, 
University System of Maryland). 

However, as the University System of Maryland is discovering in its current experiment to integrate 
MOOC materials into their courses (see Ithaka S+R, 2013), it is not so easy to take apart the components 
of a MOOC for selective assimilation into an on-campus course. Bishop described the significant efforts 
required to "unbundle" MOOCs so that faculty could, for example, embed the videos in their regular 
learning management system. 

Developing common courses to offer across institutions 

Lawrence S. Bacow, President Emeritus, Tufts University, MA, is concerned that MOOCs may only add to 
the already high costs of higher education. He believes that institutions that are genuinely interested in 
lowering these costs should focus on how to use MOOCs locally rather than worry as much about the 
"student sitting in Timbuktu." He and others suggest a strategy for sharing resources. 

Another potential saving for institutions: to offer a richer mix to students without having to staff 
up. For example, offering different languages like Pashtun or Urdu by pooling resources of 
various institutions and providing a common language course online. (Bacow, President 
Emeritus, Tufts University). 

MOOCS could save individual colleges time and money by avoiding the recreation of the same 
courses, especially developmental courses that help students prepare themselves for credit- 
bearing courses. (Online Learning Coordinator, Large California Community College). 
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If multiple institutions create content and make it available to others with a Creative Commons 
license, then there is potential for savings. But if everyone is creating the same thing we are 
losing the potential for economies of scale. I am not sure why 500 institutions are all creating 
English 101. (Royal, Inver Hills Community College). 

However, Vignare at University of Maryland University College cautions that college and university 
educators are more adept at creating new content than re-using existing content effectively. She argues 
that there is already an overabundance of content and that "...we have to figure out better ways to use 
current content and focus more on the teaching and learning..." in order to improve college attainment 
for the majority of students. Burck Smith at Straighterline noted that, a number of years ago, major 
commercial providers of educational content invested a great deal of effort in developing high quality, 
online courses but found very few academic institutions willing to purchase these materials instead of 
developing their own. 

Replacing on-campus courses with MOOCs 

Lester of FutureLearn and Dr. Paul S. Peercy, Dean Emeritus, College of Engineering at the University of 
Wisconsin-Madison, suggested that MOOCs could replace entry level courses, especially those that 
already attract several hundred students. However, Lester believes that some face-to-face supplement 
will still need to be offered on-campus. Bishop at the University System of Maryland expects that 
MOOCs could replace some aspects of existing courses. 

MOOCs in their current form appear to be good at knowledge-level content delivery because 
they're more interactive than textbooks. So, why not go ahead and give up those parts of our 
large, entry-level courses, but then bring students to campus to talk about the higher level 
concepts ? At the same time, eliminating redundancies across courses may result in some cost 
savings. (Bishop, University System of Maryland). 

Griffiths at Ithaka S+R expresses lower confidence in MOOCs as a substitute for introductory courses, 
but entertains the possibility that they could replace some less populated senior classes. 

There is also the question of what kinds of courses MOOCs are most appropriate for. Most of the 
work we have done has been around large introductory courses which are expensive to provide 
as they are often taught in several sections by tenured faculty and are often a bottleneck for 
students. But I am not sure this is the best place for them because they are such an important 
building block in a progression. On the other hand, there may be a place for them in senior-level 
classes that are hard to fill, especially in small departments with few full-time faculty. There is 
also the potential to embed modules of content in introductory level courses, more so than 
MOOCs as complete courses. (Griffiths, Ithaka S+R). 

There has been fear among many faculty members that MOOC master courses created by professors at 
elite universities to be used by other universities and colleges will make their jobs redundant. Acemoglu, 
Laibson, and List (2014) argue against the notion of MOOCs creating "winner-take-all superstar 
teachers" (p.l). Based on a theoretical model of human capital accumulation they predict that web- 
based education technologies (including MOOCs) will result in more equal distribution of educational 
resources. Their model is based on the assumption that scalability of MOOC resources created by skilled 
faculty members can be exploited only if the technology is complemented with face-to-face discussions 
with instructors. As the face-to-face instructor is freed from lecturing and allocates more time to other 
teaching activities, the quality of overall instruction will increase, thereby benefiting students. In 
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addition, if this shift helps teachers become more effective and productive, it may have a positive 
impact on the wages of less skilled lecturers. 

Faculty time savings 

Over a period of several years, Peercy at the University of Wisconsin-Madison oversaw a shift to more 
online content in a flipped classroom model for the University's engineering department. He has no 
doubts that there have been time-savings involved for course instructors which could be realized using 
MOOC materials for the same purpose. 

Let us take a three-credit course taught by a highly paid professor. ..typical numbers are: two 
hours of preparation, Monday, Wednesday, Friday; one hour of lecture, Monday, Wednesday, 
Friday; two hours of office hours, Monday, Wednesday, Friday; two hours of discussion time, 
Tuesday, Thursday. Add those hours up: 17 hours. Now put that course online. No lecture 
preparations: that saves you six hours. No lectures: that saves you another three hours. No office 
hours: that saves you another six hours. The debate students have is whether they need two 
face-to-face sessions with you or three. ..we did three. You have taken the faculty content 
workload from 17 hours a week down to three. Now what does the faculty do with those other 
14 hours? Some volunteer to develop and teach another course. ..they can spend the time in 
consulting, writing proposals, doing research. (Peercy, University of Wisconsin-Madison). 

Bacow, President Emeritus of Tufts University, offers a different perspective on flipped classroom 
models, suggesting that while they may improve learning outcomes, they do not lower costs. 

If you think about the educational production function, the most efficient part of the production 
function is putting one faculty member in front of 300 students. What is hideously expensive is all 
of the sections to ensure that students are learning something. If you just eliminate the lecture 
by flipping the classroom but create more sections, now students are watching the lecture online 
but you don't reduce the costs. One of the hypotheses in the flipped classroom model is that if 
you take the professor out of the lecture hall the professor has more time to engage students. 

But you cannot have meaningful conversation with 300 students. This necessitates more 
sections. 

Sections are really expensive. Firstly, they reguire lots of space; secondly, lots of people 
to teach the; thirdly, the need to staff large number of sections of introductory courses causes 
many institutions to over-scale the size of their graduate programs in that field. TA-ships 
subsidize graduate education. If we count the full costs of graduate education, sections become 
exceedingly expensive. The second order conseguence of this is that if we subsidize the 
production of a good, we end up over-producing it. We will end up with more doctoral students 
in a given field and wind up with diseguilibrium in the marketplace. The big cost savings would 
come when we can use technology to reduce the number of sections. That is only going to 
happen when we develop interactive education programs which do a good job of substituting 
what goes on in a section. (Bacow, President Emeritus, T ufts University). 

Reducing the need for facilities 

Bacow suggests that MOOCs and other online offerings can lead to savings in facilities expenditures, in 
particular by incorporating virtual laboratories. Armando Fox, Professor, Electrical Engineering and 
Computer Science Department and Faculty Advisor to the MOOCLab, University of California, Berkeley, 
provides a specific example of how this may play out in practice. 
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Anant Agarwal has done some interesting things with virtual laboratories in the circuit and 
design field. In addition to sections, labs in scientific fields are very difficult to maintain and staff. 
Reducing labs is a way to reduce costs. (Bacow, President Emeritus, Tufts University). 

We have 240 students this semester. We're doing an experiment where the lecture hall for the 
class only holds about 100, because we know that, after the third or fourth week, people stop 
coming to lecture because they know the lectures are going to be posted on the Internet. We just 
took the stigma away; we said, "Look, if you don't want to come to lecture, don't come. You 
won 't all fit anyway. " There are people who like coming to lecture. The lecture hall is still full: 100 
people, pretty much, are showing up, out of the 240 who are registered in the course. The rest of 
them are, I assume, watching at home, or maybe they're not watching at all. (Fox, University of 
California, Berkeley). 

Recruitment efficiencies 

Lester of FutureLearn believes that MOOCs can act as a cost-effective alternative to recruitment agents, 
especially in attracting international students. 

What's key is there's cost-effectiveness from a recruitment point of view and from a learning 
productivity point of view. The one for recruitment is straightforward. If you can build the 
systems right, and you can track people from one to the other, then you can see whether there's 
a return on investment, whether it's effective to use this as a recruitment device. If you want to 
attract international students right now you have to use recruitment agents. (Lester, 
FutureLearn). 

While the MOOC2Degree case (see Case 4 ) illustrated earlier offers a feasible mechanism for 
recruitment via MOOCs, it has not been determined whether this is cost-effective. 

Less costly student support services provided by non-faculty members 

Daniel suggests that student support functions for online courses, including MOOCs, can be provided by 
personnel who are less costly than faculty members. He suggests that savings could be achieved by 
using already existing MOOCs and outsourcing personnel. 

Randy Best atAP [Academic Partnerships] thinks that universities can deliver online courses at 
half the cost they are delivering face-to-face courses now. Online courses can help reduce costs 
of student support. AtAP they bring in part-time folks for tutoring. For MOOCs, most of the cost 
is for development, according to Tony Bates. The ratio needs to change with reduced cost of 
development and increased cost of student support. This can be done by offering credit for 
already existing courses, like those offered by MITx that are designed and ready to use, in 
partnership with organizations like AP who already have the infrastructure needed for student 
recruitment, support, and evaluation. (Daniel). 

A partial outsourcing model has been pursued for the MOOC-based Online M.S. in Computer Science 
program that Georgia Tech is developing with Udacity (see Case 5) . Each course is staffed by a Georgia 
Tech instructor and a Head Teaching Assistant (TA) serving an estimated 127 students in Year 1, rising to 
456 in Year 3 (based on projections from Spring 2013, although the actual average course size for the 
five courses offered in Spring 2014 was 115 students). The Head TA, while selected by Georgia Tech, is 
employed by Udacity. In addition, Udacity-based Course Managers, also approved by Georgia Tech, help 
students with practical issues, each serving 390 - 395 students. Georgia Tech graduate students, serving 
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40 students each, work as TAs to help with grading and responding to academic questions. Georgia Tech 
and Udacity are working on ways to manage the workload, including more use of automatic grading, as 
enrollments increase. Courses are initially taught by a faculty member, but re-runs may be taught by a 
non-tenured instructor or the Head TA who is expected to be a full-time professional and expert in the 
relevant field. It is not clear yet whether this student support model will be sufficient. According to 
Dymerska, in the Academic Partnerships model for online courses, each Virtual Teaching Assistant (the 
equivalent of a Head TA) works under the direction of the faculty member of record and serves 60 - 75 
students at most, an average of two sections, with additional "retention specialists" providing 
supplementary counseling and mentoring. 

Increasing student throughput 

Several interviewees suggested that if students can earn credit for completion of a MOOC, the possibility 
of eliminating bottlenecks in over-subscribed courses may help accelerate student "throughput." 

Another big opportunity in reducing costs: we found in our "Barriers to Online Learning" paper 
that a big motivation for some institutions to be able to offer online learning is to enable 
students to take courses that they are closed out of. As a result, their time-to-degree increased. If 
you can use online education to improve the throughput of students, the cost of education will 
fall dramatically. However, it does not show up in some conventional measures of cost. If you 
look at tuition it would not show up but if you look at cost per degree granted it is there. Lots of 
students who cannot matriculate have to stay for an extra year and are held out of the job 
market. So there are increased opportunity costs. If you use technology to broaden access to 
gateway courses there is a chance to reduce costs. (Bacow, President Emeritus, Tufts University). 

A small number of community colleges are experimenting with MOOCs as a way to lower costs to 
students and improve student retention by reducing the number of developmental courses that 
students must complete before being eligible to enroll in full, credit-bearing courses. 

The goal is to allow students to bypass classes that cost money but do not give transfer credits. It 
also takes you a year to get to college level wasting time and financial aid money. (James, Mt. 

San Jacinto Community College). 

While these colleges may appear to be sacrificing revenues by offering free, online courses to help 
students place out of fee-earning developmental courses, they expect both students and institution to 
benefit in the long run, especially as community college funding moves towards a performance-based 
model rather than depending primarily on enrollment numbers. 

...there is some research out there that says that if we can get them through developmental or 
to bypass developmental, we'll recoup that in serving that student for a longer period of time in 
curriculum classes. (Kalbaugh, Wake Technical Community College). 

MOOCs as a Source of Revenue 

While a small number of interviewees (less than five percent) speculated that some institutions of 
higher education are pursuing MOOCs as a potential source of revenue, only two representatives from 
state universities explicitly stated revenue generation as a goal. Most institutions, at least for now, 
appear to be treating MOOCs as an investment. 


Hollands & Tirthali: MOOCs: Expectations and Reality 


May 2014 


81 


Overall, the expense of engaging in this activity and developing the content is something that's 
seen as a strategic investment for our future. (Hathaway, University of Oklahoma). 

We decided early on that revenue generation would not be a primary reason [to do MOOCs], We 
believe that, long-term, this has to find a way to be sustainable as a technology investment. It 
would be great if it turns out to generate some revenue, but nobody at the campus leadership 
level believes that this is going to be a major revenue generator, or that this is the golden 
unicorn that's going to swoop in and solve the university's financial pressures. If anything, for the 
first two to four years, we assumed that this is an investment: that this is going to be a revenue- 
negative activity because we have to train faculty how to do it. (Fox, University of California, 
Berkeley). 

Potential current and future sources of revenue mentioned by interviewees included: 

• Offering credit and charging tuition 

• Creating new courses and programs 

• Drawing MOOC participants into full-tuition degree programs 

• Increasing class sizes 

• Licensing fees for use of MOOC materials by other institutions 

• Fees for additional services 

• Grant revenues 

• Matchmaking for employers 

These ideas are each described in greater detail below with examples provided of instances in which 
revenues are already being generated or are expected. 

Offering credit and charging tuition 

There are a few instances in which MOOCs or MOOC-like courses have been offered for credit and 
tuition is charged. For example, the SJSU+ Augmented Online Learning Environment pilot courses at San 
Jose State University were offered as free, non-credit-bearing courses to the public, and to a smaller 
number of formally enrolled students who earned credit and paid $150 per three- to five-credit course. 
Georgia Tech's Online Master of Science in Computer Science (OMSCS) program, launched in Spring 
2014 was due to charge $134 per credit. In each instance where the courses are offered for credit, the 
offering institution has provided some degree of supplementary face-to-face instruction or online 
support for course participants. However, based on our review of the financial and enrollment 
projections for Georgia Tech's OMSCS program (see Case 5) , we do not expect that that the revenues 
will exceed the high fixed costs of MOOC development over the initial three year period. We did not 
have access to financial projections for San Jose State University's MOOC initiative that would allow a 
similar review. 

At the University of Oklahoma, 20 credit-bearing, online courses have been developed with on-campus 
students as the primary audience in a flipped classroom model. These courses carry the same credits as 
the traditional courses and charge the same tuition. The courses are also open as MOOCs to any 
member of the public who pays no tuition, receives no credit, and does not attend on-campus sessions. 

According to Mintz at the University of Texas System, the Austin campus will experiment with a similar 
strategy later this year: 
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Among the strategies that we will explore is a value-added model that will provide enrollees with 
more support from faculty and teaching assistants. (Mintz, University of Texas System). 

Straighterline offers a useful working model for online course revenue generation whereby students pay 
a $99 monthly fee for access to any course. All courses carry ACE credit recommendations so that 
students can apply for transfer credits at degree-granting institutions and, in some cases, articulation 
agreements guarantee acceptance of the courses for credit. In addition, qualified professors can charge 
$50-$100 to provide supplemental materials and student support for a Straighterline course. Jake 
Reynolds, General Partner at Technology Crossover Ventures, points to a similar strategy employed by 
lynda.com which charges $25 - $37.50 per month for task-oriented tutorials on topics such as software 
training. 

Emily Schneider, Student Director at Stanford University's Lytics Lab, suggests another possible model of 
income generation for MOOCs that is used by Spotify, Netflix, and other entertainment services where 
consumers are willing to pay small amounts of money over time to have access to a wide array of 
materials. Schneider cautions that it will take a concerted effort among universities and platform 
providers to wean the public away from the current expectation that information on the web should be 
free. If there is a common desire to keep MOOCs free for those participants who cannot afford any fee, 
Schneider suggests that public or foundation funds will be necessary to support the endeavor, possibly 
supplemented with "some sort of tip-based, Kickstarter-style model , or pay-what-you-will at or above a 
recommended price." 

One senior administrator at a large state university, frustrated that a major MOOC platform provider will 
not permit partner institutions to charge for their MOOCs, suggested that his/her institution might offer 
MOOCs using the Canvas Network platform and charge $5 per participant. 

Creating new courses and programs 

Several interviewees suggested that the advent of MOOCs and their capacity to reach different 
audiences has created the impetus to develop new, fee-based online programs and courses. The best 
known examples of new revenue-generating courses and programs developed around the MOOC 
concept are the above-mentioned courses developed for San Jose State University's SJSU+ Augmented 
Online Learning Environment Pilot Project with Udacity (see Box 6 and Firmin et ai, 2013), and Georgia 
Tech's Online M.S. in Computer Science program also being developed in partnership with Udacity (see 
Case 5 ). 

At least two institutions were actively collecting ideas from MOOC participants about courses and 
programs they might be interested in pursuing. A few interviewees raised the possibility of developing 
MOOCs for the employees of large companies, similar to the "white label" courses that edX is already 
offering. 

Drawing MOOC participants into full-tuition degree programs 

We heard several anecdotal reports about increased interest in an institution's regular courses 
subsequent to a MOOC offering, but only found one documented instance in which MOOC participants 
were being drawn into an existing, full-tuition program. University of Texas at Arlington's school of 
nursing offered a MOOC through MOOC2Degree (see Case 4 above) and an undisclosed number of 
participants were in the process of applying to the university's program. 
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Increasing class sizes 


Several interviewees claimed that using MOOC materials to flip the classroom has allowed more 
students to enroll in a class. For example, at University of California, Berkeley, where the MOOC course 
model has been adapted on-campus to what Fox terms a "SPOC" (small private online class), automation 
of certain aspects of the courses has allowed class size to grow, purportedly without sacrificing quality of 
the experience for the students. 

The biggest benefit of the SPOC in our setting is that we've been able to multiply by almost a 
factor of five the number of students who can take our class. The enrollment of my course has 
gone from 50 or 60 students about three years ago to 240 now. We're not reducing the quality of 
the course, we're not reducing the students' evaluations of the course but we are 
accommodating more of them. We're getting more students through the system faster, students 
who no longer have to be waitlisted for a class they need. 

Historically, we were limited by the number of assignments we could grade and the 
number of TAs we could hire. Now the TA ratio hasn't changed but the TAs are spending their 
time quite differently. They're not grading assignments and exams because we have automation 
to do a lot of that. They're not rehashing what was said in the lecture because students can go 
watch that stuff online if they miss a lecture, or if they didn't get it the first time around. They're 
not creating finger exercises because we've included those along with the lectures; it's the same 
as the peer learning questions that we use. We put those online as well so students can review 
from them. The TAs can focus on higher leverage activities. (Fox, University of California, 
Berkeley). 

What is not clear, however, is whether these students are simply being diverted from other programs at 
the University, with no net increase in the numbers of students served by the institution as a whole, and 
hence no overall revenue increase. 

Licensing fees for use of MOOC materials by other institutions 

We came across several instances in which one institution had obtained a license to adopt all or parts of 
a MOOC created by another institution. However, because these were mostly experimental situations, 
no actual fee was exchanged. We only heard an actual dollar amount associated with such licensing 
from one university: 

The University System of Georgia office has a contract with Coursera that provides pricing for 
what it would cost if our faculty want to use the materials from one of their MOOCs. It is scaled 
by volume so the price actually goes down as the volume goes up. It is around $30 a 
student. (Renick, Georgia State University). 

It appears that the time and effort required to negotiate licenses for small numbers of users may not be 
worth the amount that can be feasibly charged. A licensing strategy may become worthwhile only at the 
level of mass use, for example, the recent agreement between the Kingdom of Jordan and edX where 
the former has contracted to pay a licensing fee to offer edX courses on the Arabic MOOC portal, 

Edraak. 
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Fees for additional services 

While Coursera's Signature T rack certification is already a source of revenue for both Coursera and for 
its partner institutions (as is presumably the case with edX's XSeries Certificates), the level of uptake will 
need to increase substantially to exceed the costs of developing and delivering MOOCs. The American 
Museum of Natural History found that less than 1% of enrollees (or 20% of completers) in its MOOCs 
signed up for Signature Track. However, Pang Wei Koh, Director of Course Operations at Coursera, notes 
that Yahoo recently announced it would reimburse any employee enrolling in Signature Track courses. 
We expect that the uptake for Signature Track and XSeries certificates will increase as soon as more 
employers confer economic value on them by considering these credentials in hiring and promotion 
decisions. 

Other services are being considered such as tutoring, proctoring, and supplementary materials. 

Through our partnership with Coursera, we are experimenting with a couple of things. One is our 
Pearson VUE Centers where you take your GMATs and GREs. They've struck a partnership with 
Coursera to provide some certification or offline testing. (Rizvi, Pearson). 

Vuchic, formerly at the William and Flora Hewlett Foundation and a proponent of Open Educational 
Resources, while arguing that content should remain free, suggested various supplemental services that 
can be offered for fee: 

The model that grows the fastest, as we have seen in past years on the Internet, is when content 
is free and you monetize various things around it. If you charge at the point of content then 
suddenly every user has to go through a transaction to even engage in your system. Then you 
lose 80% of your users. You can charge for certification, services, data access, dashboards; you 
can charge a classroom fee for the teacher if they want reporting on quizzes every single night. 
Maybe even student-level data if they want that. (Vuchic, formerly at William and Flora Hewlett 
Foundation). 

In some MOOCs, the instructor recommends a self-authored textbook, a controversial strategy among 
students, and proponents of open educational resources. 

Grant revenues 

Several interviewees believe that MOOCs will improve eligibility for grants. In some cases, MOOCs are 
expected to act as a dissemination route for research, helping to satisfy a frequent requirement of 
funding agencies for grantees to demonstrate broad impact beyond the immediate participants in the 
research. For example, if a new curriculum for K-12 teachers was designed based on the latest research 
findings in a particular discipline, a MOOC could be used as a form of professional development to 
provide a large number of teachers with a scaffolded opportunity to implement the curriculum in their 
classrooms (Keyek-Franssen, University of Colorado System). In other cases, MOOCs serve as a target of 
research. 

We would like to put on MOOCs as a way to disseminate research products and then measure 
the reach the research product has as a result to show granting agencies what we're doing. It 
would be great to measure the increase in grant funding that we received based on this 
innovative way to get our research product out to the larger community. (Keyek-Franssen, 
University of Colorado System). 
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A large part of what we are doing is a research enterprise and there are funding sources to 
support that. That won't cover everything and is not a long-term model but has moderate 
sustainability across time, like the OLI initiative. (Collier, Stanford University). 

Matchmaking for employers 

While we did not come across any colleges or universities that were actively pursuing a match-making 
system with employers in which the employers would pay for some of the costs of MOOCs in exchange 
for connections to strong performers, a few interviewees hinted that this might become a possibility. 

Overall, it appears that revenue streams for MOOCs are slowly building but we do not expect the costs 
of MOOC production to fall significantly given the highly labor-intensive nature of the process. While 
these costs may be amortized across multiple uses and multiple years, they will still be additive costs to 
the institutions creating MOOCs. Unless these MOOC producers can offer credentials of economic value 
in order to attract fee-paying participants, or can use MOOCs to replace traditional offerings more 
efficiently, most likely by reducing expensive personnel, they will not be able to afford ongoing 
participation in the current MOOC experimentation. Free, non-credit bearing MOOCs are likely to 
remain available only from the wealthiest institutions who can subsidize the costs from other sources of 
funds. 
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Case 5: 

Georgia Tech’s Online Master of Science in Computer Science Program 

In 2012, Georgia Tech and Udacity announced the Online Master of Science in Computer Science 
(OMSCS) Program which would allow students to earn a degree for around $6,000 instead of the typical 
$45,000. An infusion of $2 million in private funds from AT&T is subsidizing the start-up costs. AT&T 
employees were encouraged to apply for admission and made up 20% of the initial student population 
in Spring 2014. The originally anticipated expenses and revenues for the program are publicly available 
online and indicate that revenues from the initiative should exceed costs by Year 3. While noting that 
both Udacity and Georgia Tech have made it clear that these financial and enrollment projections were 
speculative, and that actual plans are constantly evolving, we replicated these financial projections using 
the published assumptions in order to assess whether the model appears feasible as a mean to reduce 
costs and/or increase revenues for institutions of higher education. 

According to preliminary staffing plans from Spring 2013 (summarized in Table 3 below), a faculty 
member assisted by a Course Developer (for initial offering) or Head Teaching Assistant (TA) (for re-runs) 
will develop and teach each of the 20 courses planned for the program. In these initial plans, projected 
enrollments rise dramatically from 1,780 in Year 1 to 24,611 in Year 3. For the first year, eight new 
courses would be developed, each serving 127 students. Six new courses would be added in Year 2 with 
each course serving 282 students. Six more would be developed in Year 3, each serving 456 students. 
Once a course is developed, it will be re-offered three times each year in Spring, Summer, and Fall. In 
Year 3, for example, six new courses would be offered and 48 re-runs. In practice, course development 
has progressed faster than expected, and enrollment growth is being managed conservatively. Twelve 
courses are expected to be complete by Fall 2014, and four more by Spring 2015. 

After the first offering of each course, either the faculty member or a non-tenured lecturer will teach the 
re-runs along with a Head TA, selected by Georgia Tech, but employed by Udacity. In addition, Course 
Managers (CMs) at Udacity will provide technical and logistical support, at a CM to student ratio of 
around 1:390. Quality Assurance staff and managers will be Udacity employees. While auto-grading will 
be used wherever practical for course assessments, some classes require group projects that were, in 
the Spring 2014 semester, hand-graded by the instructor and graduate student TAs. Work is underway 
to optimize the grading of such projects. Courses are likely to be "refreshed" every three years on 
average. According to White, rapidly changing topics such as computer networks might need updating 
every year while more gradually changing fields such as theory and algorithms may last several years. 

With respect to the financial projections, we replicated the published spreadsheets, corrected a few 
minor computational errors, and added a very conservative estimate of costs for the Head TAs/Course 
Developers which appear to have been omitted (we assumed $10,000 per course, but if these 
individuals hold an M.A. or Ph.D. as planned, the cost will be much higher). These projections also do 
not include the graduate students serving as additional TAs in Spring 2014. We summarize projected 
program costs and revenues in Table 4, showing both the published numbers and our revisions. We 
stress that we did not conduct a prospective cost analysis of this program ourselves, only a review of the 
listed resources, cost assumptions, and projections. We do not therefore purport to be verifying the 
original estimated costs. The projections show an average cost per course of $226,000 - $284,000, but 
these numbers include both new courses and re-runs. While re-runs will not have development costs 
associated with them, as class size increases they will require additional instructional support staff. 
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Table 3: Preliminary Instructional and Support Staffing Plans for Georgia Tech's OMSCS Program 


Instructional Staff and Support for OMSCS courses 

Year 1 

Year 2 

Year 3 

Projected total number of students enrolled in this year 

1,780 

10,147 

24,611 

Total no. of courses offered this year (new + re-runs) 

14 

36 

54 

New courses/Re-runs 

8/6 

6/30 

6/48 

Projected average number of students per class 

127 

282 

456 

Faculty members/lecturers 

14 

36 

54 

Head TAs (1 per re-run course) 

6 

30 

48 

Course Developers ( 1 per new course) 

8 

6 

6 

Number of Udacity Course Managers (CMs) 

5 

27 

63 

Ratio of Students to CM 

391 

390 

395 

Average number of CMs per course 

0.4 

0.75 

1.2 

Ratio of Students to Quality Assurance staff member 

1,953 

1,977 

1,951 

Ratio of students to manager of CMs and Quality 
Assurance staff 

6,509 

6,590 

6,505 


Sources: David White, Assistant Dean for Academic Programs and Executive Director of the Online MS in Computer 
Science, College of Computing, Georgia Tech ; Online Courses Hosting Agreement Georgia Tech and Udacity. 

Total projected program costs rise from around $3 million in Year 1 to around $15 million in Year 3 
(using our revised estimates). If the enrollment projections hold, this will allow the costs per 36-credit 
degree to fall from an initial $12,820 to $4,212 by Year 3. Assuming revenues of around $150 per credit, 
this will yield a net loss for the program in Years 1 and 2, but a net gain of $34 per credit in Year 3 
(excluding the university's technology fee and the $2mm from AT&T). However, we consider a number 
of assumptions and plans in the projections to be at risk: 

• The expectation that 24,000+ students will be enrolled in Year 3 appears highly optimistic. 

• The expectation that dropout rates will be as low as in Georgia Tech's traditional program is not 
supported by data from other online programs where dropout rates are higher than for on- 
campus students. 

• The very high ratio of students to Teaching Assistant and Course Manager in each course seem 
unlikely to provide adequate support for either the lead instructor or the students. 

• The plan to develop only 20 courses in total and to re-run each one three times per year 
following their creation does not allow much choice to students who need 12 courses to 
complete the degree, and assumes that new students can be recruited into courses that are re- 
run every semester. 

• More generally, the financial plans, even if they pan out as expected, do not accommodate any 
significant contribution to the indirect costs of Georgia Tech as a whole, with each student 
paying a fraction of the tuition paid by regular students. 

While such a program could serve as a loss leader for the institution, attracting publicity and approval 
from policymakers, most institutions can ill afford the running costs of such initiatives. We suspect that 
AT&T will be a greater financial beneficiary of the program than Georgia Tech itself, having essentially 
arranged exceptionally cheap training for its employees through a tax-deductible contribution. 
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Table 4: Summary Projected Costs and Revenues for Georgia Tech's OMSCS Program 

Non-highlighted items were derived from the published financials. Highlighted entries show our own 
recalculations with minor computational errors corrected and the addition of a Head TA/Course 
Developer that we believe was mistakenly omitted. 


Estimated/projected item 

Year 1 

Year 2 

Year 3 

Projected total number of students enrolled in this year 

1,780 

10,147 

24,611 

Total no. of courses offered this year (new + re-runs) 

14 

36 

54 

New courses/re-runs 

8/6 

6/30 

6/48 

Projected average number of students per class 

127 

282 

456 

Estimated total costs of program 

$3,138,800 

$7,520,200 

$14,399,700 

Head TA/Course Developer added at $10, 000/course 

$3,375,165 

$8,127,983 

$15,311,332 

Cost per course offered (including new + re-runs)* 

$224,200 

$208,894 

$266,661 

Head TA/Course Developer added at $10, 000/course 

$241,083 

$225,777 

$283,543 

Estimated cost per credit to GT and Udacity combined 

$331 

$141 

$110 

Head TA/Course Developer added at $10, 000/course 

$356 

$152 

$117 

Estimated cost per credit to Georgia Tech only 

$104 

$35 

$19 

Head TA/Course Developer added at $10, 000/course 

$129 

$47 

$26 

Student services cost per credit 

$45 

$45 

$45 

Total degree cost for 36 credits 

$11,920 

$5,072 

$3,952 

Head TA/Course Developer added at $10, 000/course 

$12,820 

$5,472 

$4,212 

Revenue per credit from tuition + fees** 

$145 

$149 

$151 

(Loss)/profit per credit** 

($186) 

$8 

$41 

Head TA/Course Developer added at $10, 000/course 

($211) 

($3) 

$34 


*The published numbers do not allow for a separation of costs of re-runs from new courses. 
** Excluding revenues from technology fee and the $2,000,000 grant from AT&T in Year 1. 


Source: http://s3.documentcloud.oro/documents/703593/udacity-otrc-amendment-5-13-2013.pdf 
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Goal 4: Improving Educational Outcomes 


Thirty-eight percent of the institutions participating in our study that were offering or using MOOCs, and 
twenty percent of our interviewees overall, expected MOOCs to lead to an improvement in educational 
outcomes, some directly within the MOOC format and some indirectly through the transfer of new 
strategies and techniques to on-campus teaching. This goal was mentioned by representatives of five 
public universities, four private universities and two community colleges. Ironically, most of these 
interviewees were at institutions that would be considered among the elite. 

My real issue is teaching people. I know how to teach people in my research group to be experts. 

I am now seeing how un-expert our whole undergraduate system really is, how we teach 
students to do problems in a certain format by some special shortcut way, where they don't 
really understand it as we think they do when they do it. I want to really try to learn how to 
teach more people to be experts. And I want to use the MOOC to do it, to accelerate the process. 
(Pritchard, MIT). 

Professor Sanjay Sarma, Director of Digital Learning at MIT, expresses concern about current teaching 
practice and optimism for the role MOOCs can play in rethinking it. 

We have looked at Eric Mazur's research about brain activity during class and see that the way 
we are teaching now may not be the best. In some ways MOOCs are a Trojan horse to help our 
colleagues rethink the way they teach. We have to deal with some cultural barriers, for example, 
professors think lectures are pretty good. (Sarma, MIT). 

Sarma reported that one of MIT's goals for using MOOC resources and technology is to allow students to 
come into class more prepared so that instructors can re-deploy attention to one-to-one teaching. Fox, 
at University of California, Berkeley, concurs that MOOC materials can be used to optimize instructor 
time. 


If you can create materials in a way that instructors can spend more of their time on the things 
that differentiate them, working with individual students or adding some of their own 
supplementary materials and ideas to provide a gentler on-ramp to a subject, that's a better use 
of their time. (Fox, University of California, Berkeley). 

Interviewees mentioned several ways in which MOOCs may lead, directly or indirectly, to improvement 
in educational outcomes: 

• Motivating instructors to rethink pedagogy 

• Course re-design 

• "Chunking" lectures and interspersing questions 

• Fine-tuning instructional materials 

• Providing instant feedback 

• Gamification and badging to increase motivation 

• Outreach to participants to encourage persistence 

• Adaptive learning/personalization/mastery-based learning 

• Using MOOCs in K-12 to prepare students for college 

We elaborate on each of these strategies below, providing examples of where they have been used. 
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Motivating instructors to rethink pedagogy 

For a number of interviewees, the most significant impact of MOOCs has been on the motivation they 
have created for instructors to rethink how they teach. As Siemens observed, "Regardless of format, the 
passion of teaching is ignited for faculty teaching MOOCs." In addition to students learning the specific 
content, methods, tools, and skills of the academic discipline on which a MOOC is based, additional 
objectives of cMOOC instructors included providing an "atelier" environment where students and 
instructor work together in “a social and technical network" to create artifacts, and a permanent digital 
space to store creations. 

Several interviewees reported that MOOCs have prompted even typically intransigent faculty members 
into reconsidering their teaching styles and have facilitated faculty professional development, which 
might otherwise be resisted. 

A lot of the professors have not had to teach students of such diverse backgrounds before so 
they probably think about audience and what is needed for the students to be successful more 
deeply than ever before. This is common for online courses in general. I have been in faculty 
development for 10 years now and we call this "guerilla-style" faculty development because 
when you ask a faculty member to teach online, it is often the first time they think about their 
course design. Because they are willing to get input on the technology-related issues, you can 
subvertly throw in some ideas about course design, addressing different student audiences, 
writing learning outcomes. But you can see with MOOCs that faculty are really thinking about 
what they want students to get out of the experience, especially as they are not being motivated 
by course credit. "How do I motivate them, how do I think about who they are, what does it 
mean to be successful, what will they take away from this? I think they now think differently 
about even their on-campus, Stanford students." (Collier, Stanford University). 

Guthrie at ITHAKA also believes that instructors are changing the way they design and deliver courses as 
a result of their experiences with MOOCs, and has no doubt that instructors feel that their teaching has 
improved, but he points out a lack of evidence that better learning experiences are actually being 
created for students. As he illustrates, change in practice is not inherently positive: 

If I sit down to write a speech and I have to write down every word, I have to do a very different 
thing than when I just stand up and start talking off the cuff. I have to think about it much more 
carefully; it's much harder work. That doesn't necessarily mean my speech is better, but I 
definitely understand the material better, I have to frame it very carefully. I think faculty 
members are having that kind of experience when they shift over to delivering a MOOC. 

(Guthrie, ITHAKA). 

Course re-design 

Many interviewees recounted that faculty member engagement in MOOCs has led to re-design of their 
courses. Ugoretz at CUNY observed the inherent capacity for online experiences to provide instructors 
with feedback on their teaching: 

One of the things I find most powerful about online learning is that, for the teacher, the usually 
ephemeral information is concrete, and it's right there. You can look back at it to modify your 
teaching. I've walked out of a face-to-face class thinking "That was a great class!" or "That really 
didn't go well. " I may remember some of it, but I don't remember exactly where it went wrong. 
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With online discussion, you can go back and you can see, "OK, here's where I said the wrong 
things, " or "Here's where I should have steered us back, or let it go further in the other 
direction." (Ugoretz, Macaulay Honors College). 

Often, design changes are made initially to conform to the typical xMOOC format of multiple, short 
videos with embedded questions, weekly quizzes, and assignments. However, many instructors have 
found that the process pushed them to reconsider how to present concepts and materials. They have 
also discovered that some of these changes can be adapted for on-campus courses. Sarma at MIT 
promotes the concept of "modularity," noting that shorter courses are both easier to create and to 
complete. Courses created for the edX platform have generally been shorter than on-campus classes 
and have led to the "modularization" of some on-campus courses. 

Box 4, Case 6, and Case 7 provide examples of how on-campus courses were restructured as a result of 
the instructor either teaching a MOOC or integrating one with his class. The most popular strategy we 
encountered for re-structuring instructional time was "flipping" the classroom, a concept that 
significantly pre-dated MOOCs. MOOC material is made available for students to work on independently 
while face-to-face instructional time is optimized by engaging in problem-solving and discussion. For 
example, Harper at the University of Oklahoma re-designed his on-campus course Law and Justice 
putting short videos, quizzes, an interactive textbook, and discussion components online, while in-class 
time was focused more on discussion and debate. 

Another common, but pre-existing, strategy being introduced in many courses is peer learning. Guthrie 
at ITHAKA believes that the most powerful learning innovation that MOOCs present is the ability for 
"many to many " communication and that leveraging this capability may produce the most learning gains 
for participants. Siemens notes that large enrollment numbers permit self-organizing "sub-clusters" of 
interests and introduce a broad variety of interests. The University of Oklahoma has been working with 
NextThought to deliver online courses that exploit the potential of online communities to foster socially- 
constructed learning (see Case 8) . 

Experimentation with large-scale peer assessment techniques have led to improvements in peer-to-peer 
rating strategies with rubrics developed and iteratively revised to maximize clarity and consistency of 
grading. James at Mt. San Jacinto Community College helped design the college's Crafting an Effective 
Writer MOOC which included three assignments that were reviewed by peers and a culminating 
assignment that was peer-graded. She observes that "The rubric has to be incredibly detailed for it to 
work." Some interviewees were less sanguine regarding the current status of peer grading and its 
appropriateness. For example, Michael Cennamo, Educational Technologist, Columbia Center for New 
Media Teaching and Learning (CCNMTL), Columbia University, remarked: “We are not ready to take that 
idea to thousands with no credit, no risk, no liability." 

A number of interviewees suggested that the integration of MOOC materials created by other faculty 
members into their courses, a concept often termed "wrapping" or "embedding" a MOOC, in a fashion 
parallel to the use of textbooks, will eventually become more accepted practice and will benefit 
students (see Fisher, in press). Fox at University of California, Berkeley coined the term SPOC (small 
private online courses, see Fox, 2013) to describe one version of this application of MOOCs. 

SPOCs are a big part of the equation and are not getting enough airplay. This is going to change 
the way we think about curricular technology transfer. As instructors, we don't all use the 
textbooks that we wrote. Most of us use other people's textbooks. We don't feel that we have 
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abdicated control of the course just because we didn't write the textbook. It's taken us a fair 
amount of time to get acclimatized to that point of view, but I think we've got to acclimatize to 
this one too: that if you use somebody else's MOOC materials, that doesn't mean that you've 
given up your control of the course. It means their materials are really good and you think you 
can do a better job helping students if you work with those materials than the ones you had 
before. That's an unsung but really interesting opportunity. I don't think it's going to be free, but 
I'd like it to be very cheap for an instructor to just click the clone button on a MOOC and say, "I 
want a version of this to use in my own classroom. " (Fox, University of California, Berkeley). 


Box 4: 

Rethinking the Design and Delivery of Engineering Software as a Service at UC Berkeley 

“ When we did the 'Engineering Software as a Service' MOOC the first time around, we were advised by 
our colleagues, 'Don’t just record your hour-and-a-half-long lecture.' The hour-and-a-half-long lectures 
from previous runs of my course have been online for years. But my colleagues had pioneered a format 
of 10-minute segments with freguent self-assessment guestions for the students. We thought that if we 
were going to do the work to create these guestions, we should find a way to have the on-campus 
students benefit from them as well. We decided to use peer learning. We restructured the lectures on- 
campus with the understanding they were going to be captured live and edited after the fact. Now we 
talk for 7-10 minutes and then we stop and ask a question. The students discuss it with their peers, and 
they vote on the answer. This has kept the students much more engaged in lectures, so much so that 
even if I never taught another MOOC again, I'm going to keep doing my lecture-planning this way. It 
turned out that was a much better format for the live lectures to begin with. Now, you could argue that 
the peer learning literature has said for years that this is a good idea. So why do you need MOOCs as a 
kick in the pants to do it? Because in MOOCs you are forced into certain design decisions that you 
wouldn't necessarily be forced into in a campus class, and the realization is that the new design actually 
would have been good for the campus class if you had the initiative to try it. 

In fact, the peer learning strategy on-campus has worked out so well that I now feel bad for the MOOC 
students because they just get a self-assessment question. The students in the classroom get a chance to 
discuss that question with their peers before they decide on the answer, which study after study shows 
helps them learn the material better. We're actually in the process of doing an experiment right now to 
see if that aspect can be brought back into the MOOC. Students could have small, virtual chats within the 
MOOC with colleagues in a virtual chat room and then answer the question together. We're going to see 
whether, in an A/B test, students who answer the question that way are more likely to not only get their 
question right, but also do better on those questions in the assessments and the quizzes. We're doing a 
dry run of this in a MOOC now. And we hope to do a production run with an in vivo experiment in the 
spring. 

Source: Armando Fox, University of California, Berkeley. 
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"Chunking” lectures and interspersing questions 

Rebecca Petersen, Director of Research and Educational Initiatives at edX, reports that early indications 
of video-watching behaviors show many MOOC participants will watch only three to five minutes of a 
video clip, regardless of the actual length of the clip. Many MOOC videos, which are rarely longer than 
10 minutes, are now interrupted at regular intervals with "inline" questions to check the viewer has 
grasped the foregoing content or can apply it to a problem. Udacity takes this strategy to the extreme 
with questions interspersed as often as every two minutes. Numerous MOOC instructors, including Fox 
at University of California, Berkeley (see Box 4) and Racaniello at Columbia University, have taken the 
strategy of "chunking" lectures and interspersing questions back into their live classrooms. 

One thing I like to try is to ask questions during my live lectures using clickers. I did not really see 
the use until I did my Coursera course. I liked the idea of pausing and making sure that everybody 
knows what is going on. So it is like a video quiz question, but live. It is a pause. In Coursera you 
break the lecture into chunks so that students can digest a chunk. So I can do that in my lectures 
too: stop and see if they really get it by seeing the results right there. It will slow things down but 
it will be better for their comprehension. I am going to do that in Spring. 

I think I am also going to be aware of thematic chunks in my lecture because of this. In fact, I was 
thinking of posting the Coursera videos. They are ours to do as we want. Right now I post 
lectures that are an hour and a half. What if I post 15-minute chunks [for the on-campus 
students]? Maybe somebody busy can watch for 15 minutes. That may help them learn it better. 

I could cut it up right away for the Columbia course and it will be all ready for Coursera. 
(Racaniello, Columbia University). 

Fine-tuning instructional materials 

In some cases, rather than re-designing an entire course, instructors are fine-tuning instructional 
materials, for example, substituting a textbook and other publisher material with open educational 
resources. Several community college interviewees noted that the move towards free texts that began 
with the Open Educational Resources movement, and is being adopted in some MOOCs, was a 
significant benefit to their students who often struggle to pay for textbooks. 

Sarma at MIT described the development of "instrumentation" to measure activity in MOOCs and allow 
adjustments to improve instruction, for example to identify concepts that students appear to be stuck 
on, as evidenced by multiple video rewinds or attempts at a question, and then revise the content to 
provide alternative explanations. But Jaggars at CCRC notes that the current turnaround time for 
analyzing most MOOC platform data is too long to allow for mid-course corrections. 

Loble, formerly at University of California, Irvine Extension, listed several adjustments being made to 
both the university's "traditional" online offerings as well as to future MOOCs as a result of participant 
feedback in the initial MOOCs. These include posting more videos of the professor; being more explicit 
in expectations and directions to students; and establishing an online "getting-to-know-you area." 
Another lesson learned was that the MOOC medium does not work for all content. 

Providing instant feedback 

Fox, who has run his Engineering Software as a Service MOOC several times and has built sophisticated 
auto-graders to assess student code, believes that the MOOC format affords students more frequent 
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opportunities to demonstrate mastery and receive instant feedback than were available when all 
assignments were graded manually. He acknowledges that there is still room for improvement. For 
example, the auto-graders are not yet capable of providing feedback regarding style of coding, as 
opposed to simply whether it is functionally correct. At MIT, Professor Michael J. Cima's experiment 
with Introduction to Solid State Chemistry, described in Case 6, is a testament to the power of regular 
and instant feedback. 

While many observers are skeptical of the applicability of auto-grading to anything but multiple choice 
or numerical assessments, Ryan Baker, Associate Professor at Teachers College, Columbia University, NY 
is more optimistic. 

I actually don't share the skepticism about computer grading: computers make different errors 
from humans but when you look at agreement between computers and humans for best 
practices text scoring work, it's actually about the same as agreement between humans and 
humans. (R. Baker, Teachers College). 

Gamification and badging to increase motivation 

Faculty members and administrators involved in designing and delivering Tri-C's Pre-Algebra CCC MOOC 
(see Case 2 ) found the MOOC format lent itself well to one of the more important tasks in serving their 
students: encouraging persistence. One way in which this was accomplished was by introducing 
elements of game mechanics into the course so that students could progress through four increasingly 
challenging levels of activities. However, as Thackaberry noted, gamification is expensive to execute. 

What we did was kind of the low-rent version of game mechanics, using Adaptive Release and 
Blackboard. Actually, it mimics some of the most successful use of educational gaming, because 
it doesn't overwhelm the user, so that the student can focus on the simplicity of conquering the 
content, rather than learning how to navigate in a virtual world. That's also good news for us in 
terms of expense. We're not Sony PlayStation. We can't spend $5 million developing an 
immersive game. (Thackaberry, Tri-C). 

While several other interviewees mentioned gamification as a strategy to engage participants in online 
environments, only two other institutions described actual examples of gamification in their courses. 

A second strategy Tri-C used to encourage persistence and "good online citizenship" in the Pre-Algebra 
CCC MOOC was the incorporation of Mozilla Open Badges . Royal, who was instrumental in the 
development of the Tri-C MOOC before moving to Inver Hills Community College, describes the 
reasoning behind incorporation of badges: 

The concept of badging is a great one, we see it in a lot of social aspects within our society and I 
think there is an opportunity to use the same principles in educational activities to help 
encourage, support, and motivate students to complete whatever their goals are for their 
educational path. It has become pretty standard for the millennial generation and people 
entering college to have digital representations of themselves: having avatars or profiles. So 
badging seems like a natural extension to that. (Royal, Inver Hills Community College). 
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Outreach to participants to encourage persistence 

A third strategy to encourage persistence in Tri-C's Pre-Algebra CCC MOOC was the implementation of 
motivational outreach and engagement activities by the MOOC instructors. According to Thackaberry, 
from one iteration of the course to the next, faculty members increased the number of special 
announcements and e-mails including "video snippets and other attention-grabbers." In some cases, 
these were directed to all students while, in others, messages were targeted to individual students who 
had become inactive in the MOOC, in an effort to persuade them to return. Miles describes these efforts 
as purely motivational rather than content-related: "We were emphasizing greater engagement with the 
content, or directing students to the content, or supporting them through the content.” Course size was 
small enough to make it possible to monitor and interact with individual students: just over 300 
students were active in the fourth and final facilitated iteration of the course. These motivational tactics 
led to an increase in the course completion rate from 12.5% in the first iteration, to 19.3% in the last, 
based on the number who completed a mandatory pre-survey. If it were possible to replicate these 
motivational strategies in larger MOOCs, it is likely that the completion rates would increase, at least for 
those participants who start with the intention to complete the course. In Goal 6: Research on Teaching 
and Learning, we report on several research efforts working in this direction. 

Adaptive learning/personalization/mastery-based learning 

One of the pedagogical attractions of online learning is the possibility of providing adaptive learning 
experiences, that is, providing a learning trajectory for each individual student that responds to ongoing 
assessment of how that student is performing. For example, if a student is struggling with a concept, the 
learning platform may direct him to a prior learning experience and only return to the original trajectory 
once he demonstrates a grasp of the prior concept. Students that master the targeted skills and content 
quickly can complete the course at their own pace without being held to a "seat-time" model of 
learning. 

Stephen Laster, Chief Digital Officer at McGraw-Hill Education, asserted that particularly for MOOCs 
which are open access, there must be recognition that participants will begin the course in various 
states of readiness and bring with them many different learning styles. He sees value in providing a pre- 
assessment that determines the prior knowledge and skills of participants. Such an assessment could be 
used to advise participants whether they are ready to take the course or to direct students to more 
scaffolded material. By allowing more accurate targeting of the course to participants, the completion 
rates may improve. Petersen at edX indicated that a few MOOC developers are indeed using pre- 
assessments for learners to judge for themselves their skill level relative to the content being presented. 
But she added that, for now, it is up to the learners to find their way through the materials; it is not the 
system that responds to a learner's performance and skips him ahead or re-directs him to polish a skill. 

Jaggars at CCRC questions whether true adaptive learning will ever be affordable: 

I'm skeptical that a for-profit company would be willing to invest in the huge amount of research 
and development necessary to create truly high-guality learning software. Most companies seem 
to invest just enough to get the product to sell, not to get the product the best it can be. They 
may think their return on the additional investment would be too small to be worth it. Some of 
the best learning software we've seen was developed at research institutions with support from 
foundations, but I'm not sure whether a philanthropic development model is sustainable. 

(Jaggars, CCRC). 
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Based on her assessment of the current state of adaptive learning environments, Griffiths of Ithaka S+R 
suggests that truly adaptive learning is still an aspiration rather than a reality: 

If 10 is the ultimate realization of adaptive learning environments to achieve their full potential 
with feedback loops and continual improvement, we are probably at a 2 with the tools we have 
now; they are pretty primitive. I don't doubt that it will happen, but we are just in the really early 
days of figuring out at what level these tools can make the most impact. With the technology we 
are using now, we can just change the sequence of the learning objects in a way that is supposed 
to reflect the way that student learns, what he already knows, and what he needs to focus on. 
That doesn't seem to make much difference to the student's instruction: they are getting 
through the same material. It is not a transformative experience. The instructor can choose to 
turn on the adaptive feature or not and it's not that big a difference. (Griffiths, Ithaka S+R). 

Using MOOCs in K-12 to prepare students for college 

A few interviewees believe that MOOCs may be successfully blended into advanced courses at the high 
school level to help prepare students for college. For example, Pritchard at MIT believes that MOOCs 
could be integrated into Advanced Placement classes, Baccalaureate programs, or "A" levels in the 
United Kingdom. There have already been a few reports of MOOC participation by high school students, 
both as part of their school work and outside of it. For example, McGrory (2013) reports that high school 
students in Florida began to participate in MOOCs after Governor Scott "signed a law allowing MOOCs in 
subject areas with end-of-course exams, including algebra, geometry and biology." According to 
McGrory, Broward College, FL is offering a MOOC on college-level math, reading, and writing with half 
the participants being high school students from Broward County. St. Petersburg College, another 
community college in Florida, has offered a remedial math MOOC in which high school students from 
Pinellas County participated. It is not clear, however, what the impact of such MOOC participation has 
been for these students. 

Participants at the recent NYSCATE New York State Online Learning Forum (February 6 th , 2014), 
suggested that while the costs to produce a MOOC would be prohibitive for regular high schools, there 
may be possibilities for existing MOOCs to be pursued by high school students in independent studies, 
and for teachers to participate in MOOCs for professional development. In New York State, school 
principals may decide whether credit can be earned for participation in an online course, but Paska, of 
Harrison Central School District, raised a number of policy challenges that limit the extent to which 
MOOCs can currently be used within existing K-12 curricula. For example, in New York State, while 
online courses may be offered by districts, they may not be used to reduce teaching staff, and the 
students must interact with a district-based teacher of record. Paska indicated that most states cap the 
teacher's student load and require that online courses be taught by a state-certified teacher. 

Doubts about the capacity of MOOCs to improve teaching and learning 

We came across several skeptics regarding the potential of MOOCs in their current form to improve 
educational outcomes. As Kolb of RPI pointed out, universities and colleges are "not only delivering 
content, but trying to create learning environments" while MOOCs are "... for the most part, just taking a 
swing at the content piece." Kolb worries that xMOOCs fail to encourage divergent thinking and 
contribute to higher education's propensity to "knock creativity out of our students.” For David 
BenDaniel, Berens Professor of Entrepreneurship at the Johnson School, Cornell University, MOOCs 
simply cannot accommodate the style of teaching he finds most effective: 
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...my teaching is entirely Socratic: back and forth, give and take with the students. It is 
something intimate with the class. I walk up and down the aisles. I stop and say "What do you 
think about this?" I cannot possibly imagine being able to "can" one of my courses where a 
discussion group can answer all the guestions that come up. You cannot duplicate in a MOOC the 
same rapport that can be created with a limited number of students in a face-to-face class. The 
use of humor is an example. (BenDaniel, Cornell University). 

Hersh, formerly at Santa Barbara City College, concurs with this view asserting that "... right now the 
best trend in learning goes back to Ancient Greece with the Socratic Method." Hersh observes that while 
MOOCs may address some of the cognitive aspects of learning, their lack of "human presence" results in 
a failure to address important affective aspects. He expects that unless MOOC designers devise 
strategies to foster a greater sense of community and provide additional supports, perhaps in the form 
of virtual TAs ora "Siri" equivalent, completion rates are likely to remain low. 

Li Fengliang, of Tsinghua University in China, does not expect MOOCs to have a significant impact on 
higher education because he doubts that they will be effective educational tools for any but those 
learners who start with a high degree of interest and ability. Dede, of Harvard, cautions that only a very 
small number of people are able to learn from instructional materials without any additional form of 
support and argues that MOOCs will only become successful on a broader scale if peer learning 
strategies are enlisted. Moreover, Dede feels that the best learning experiences will arise between the 
cMOOC and xMOOC extremes. 

Kalbaugh, from Wake Technical Community College, reacting to the recent PCAST letter to President 
Obama concerning the potential of MOOCs, similarly expresses concerns about which students may 
benefit from them. 

From where I sit, I also have grave concerns about new students; underprepared students; older 
returning, students having the ability to truly learn in such a radically different environment from 
what they are familiar with. Those are the very students who are finding postsecondary 
education difficult and who may have the most difficult time with a MOOC environment. For 
highly capable and prepared students, MOOCs may very well be a great alternative, but these 
are not the at-risk students that the PCAST letter indicates MOOCs will help. (Kalbaugh, Wake 
Technical Community College). 

Many interviewees pointed to the pilot initiative conducted by San Jose State University with Udacity 
(see Box 6 and Firmin et al., 2013) as evidence that MOOC-based courses are ineffective for at-risk 
learners. 

Dede at Harvard, Thille at Stanford, and others who have spent many years perfecting the process of 
designing and re-designing online learning experiences, lament the inattention to existing research and 
knowledge about what does and does not work in online learning: "We are starting without reference to 
history and making the same mistakes over and over again" (Dede). Thille remarked that much that has 
been learned about effective design in the past is not only being ignored, but "rediscovered." Bates at 
Contact North is, however, hopeful that with universities having prior online experience now joining the 
MOOCscape, this scenario will change: 

I'm still seeing speakers from MIT and Stanford deliberately ignoring previous online learning 
developments as if they were unimportant. This does a great disservice to the many credit-based 
online learning programs that pioneered good guality course design based on research and 
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evaluation of what works and what doesn't in online teaching. However, on a more positive 
note, I'm seeing an increasing number of MOOCs moving away from the video lecture format 
into better designed, more interactive models, as universities with prior experience in online 
learning have been getting into the MOOC market, so perhaps eventually MOOCs will find their 
true niche in the market. 

Stan Silverman at NYU suggests that MOOCs are failing to leverage technology to best effect in 
educating people: 

We are not convinced that a massive approach to anything works. You homogenize it. We're 
now talking about customizing learning. The word massive suggests you're going in exactly the 
opposite way. It contradicts the basic philosophy that we think the technology is best used for. 
(Silverman, NYU). 

Guthrie at ITHAKA points out that it requires substantial investments to realize value from the enormous 
amounts of MOOC platform data in order to improve educational outcomes: 

I think that even the platform providers are going to be struggling to figure out how to turn that 
data into analysis that can improve instruction. I haven 't heard that people have been able to 
turn that data into impactful changes yet. (Guthrie, ITHAKA). 

Bror Saxberg, Chief Learning Officer at Kaplan Inc., is optimistic that MOOCs can help provide certain 
skills but does not expect that they will help prepare participants to succeed in complex tasks in the 
workplace: 

I don't think MOOCs are any kind of answer at all to building expertise that is relevant in the 
workplace. I think zero-variable-cost solutions are enormously important at the beginning of 
your trajectory towards becoming highly capable in the workplace. But we know that, on top of 
technical skills, it takes communication skills; judgment; a whole range of complex, grey areas to 
be successful at work. That's going to reguire feedback from professionals who are already doing 
it in order to hone your skills, if the objective is to get you as ready as possible to do the work at 
an expert level when you finish your training program. So, how far can zero-variable-cost 
solutions carry you? I think in some areas like computer science and computer programming, 
they might carry you pretty far, but not all the way, because you've got to talk to business 
people. You've got to talk to other engineers. You've got to manage people to do a complex 
project. All of those things are going to need human coaching and experiences with human 
beings. (Saxberg, Kaplan Inc.). 

While interviewees provided many example of how MOOCs have been used to change instruction, for 
the most part, actual impact on educational outcomes has not been measured in any rigorous fashion, if 
at all, so it is unclear whether the goal of improving educational outcomes has been achieved. Case 6 
and Case 7 highlight two exceptions where instructional changes and their effects on student 
performance have been carefully documented. While not gold-standard, random assignment 
experiments, they provide promising evidence of improvement in student performance as a result of 
integrating MOOCs into flipped on-campus courses, or adopting some of the MOOC strategies such as 
frequent assessment and automatic feedback. In the section Goal 6: Research on Teaching and Learning 
we review more formal research that has been conducted on MOOCs. 
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Case 6: Course Re-design to a Mastery-Based Model: Michael Cima, MIT 

Michael Cima, Professor of Materials and Science Engineering at MIT, has been teaching solid-state 
chemistry on-campus for 28 years and has been instructor in charge for three years. He was invited to 
offer the course as a MOOC in the Fall of 2012 and, over a period of around six months, spent several 
hundred hours re-designing the course to conform to the current xMOOC format. Once the MOOC 
Introduction to Solid-state Chemistry was completed, Cima decided to experiment with some of the 
MOOC instructional strategies in his face-to-face course to see if he could improve learning outcomes 
for the on-campus students. He surveyed students who had taken his course in previous years to solicit 
feedback on potential changes to the course design, ranging from eliminating lectures to no change at 
all. Not surprisingly, he found that the top performers were inclined towards no change in the course 
design while those who had struggled were more enthusiastic about changing the assessment format. 
Cima's goals for the re-design project were: 

• To experiment with new forms of assessment that might stimulate more learning and more 
accurately measure what students know. 

• To ensure that all students kept up with the material from the beginning of the course rather 
than wait for the customary MIT "fifth-week warning flag" to notify students who were not 
meeting expectations. 

• To bring up the bottom of the class and raise the pass rate of the course to 100% from an 
already high rate of around 97%. 

The re-designed on-campus course was delivered in the Fall of 2013. It involved three lectures and two 
recitations per week, as in previous iterations, but a requirement to attend 80% of the sessions was 
instituted. The course text was replaced by the online content from the MOOC. This content consisted 
of "learning sequences" which students could use as much or as little as they needed. 

Each learning sequence is five minutes of me at a chalkboard, then five minutes of a screencast, 
like a Khan Academy kind of thing. Then there may be a practice problem that you can do. It 
alternates like that for, on average, about 40-50 minutes of video podcasts. It's like a lecture, but 
I think it's more interesting than a lecture, because it breaks it down into individual concepts and 
you are asked to attempt a problem related to the material the lecture's about. I actually like the 
screencast better than the lecture, because I find my tone is very different. I've been doing this 28 
years. Give me a piece of chalk and put me in front of a blackboard , I'm lecturing. If I'm doing a 
screencast, it's as if a student is standing over my shoulder. The work I do is so completely 
different. I get very positive responses to my screencasts.(C\ma, MIT). 

Cima found that putting the content online was particularly helpful in addressing the fact that students 
regularly enter the course with a wide range of background in chemistry, ranging from almost none to 
three years' worth of high school courses. In practice, he estimated that around 50% of the students 
used the online content. Course assignments and assessments were overhauled to replace homework, 
mid- and end-of-term exams with 14 weekly, online, proctored opportunities for students to check their 
understanding and demonstrate mastery. The online assessments (a total of 37 spread over the 14 
units) were constructed from problems that had been collected in a database over the previous 40 years 
for use in mid-term and final exams. Each week, students were expected to attempt problems delivered 
on a platform specially designed for the class by Cima and Isaac Chuang, MIT Professor of Physics and of 
Electrical Engineering and Computer Science. The platform presented students with problems selected 
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at random from the database. In some of the problems, even the numbers used within the problem 
were randomized to ensure different students received different versions. 

Assessment problems were not straightforward multiple choice questions that students could guess at 
or rely on the answer options for hints. According to Cima, 70% of the problems required "floating point 
answers", that is, a calculation had to be performed and the numerical answer entered online. Students 
were usually permitted up to three tries for each of these questions. Where multiple choice questions 
were used, Cima presented "tiered sequences where you have to enter all of them completely correctly 
to get the assessment done. If you submitted it with one of those things wrong, you'd have to do the 
entire assessment again.” Multiple choice questions could only be attempted once. 

Weekly assessments were completed in a proctored assessment room that had been set up for the 
course with tables, Chromebooks, and Internet drops. The room was staffed with proctors for three 
hours a day, six days a week, and had seating capacity for 40 students at a time. Assessments were 
automatically graded, providing instant feedback to students, and after a 24-hour lock-out period each 
time, could be attempted multiple times during a two-week window. Allowing multiple attempts was an 
adaptation to compensate for the fact that partial credit could not be earned for solving problems part- 
way without reaching the correct answer. Teaching assistants (TAs) were available for consultation 
before or after the assessments in a room adjacent to the assessment room, so that students could 
resolve any difficulties on the spot. Cima noted that the assessment room and TAs were used much 
more heavily than expected and that twice the seating capacity or twice the number of hours would 
have been optimal. Cima surmised that, at mid-term, the 361 students were, on average, making two 
attempts at each of the assessments. 

Cima was astounded at the increased effort that students made in practicing problems and re- 
attempting assessments in order to satisfy the course requirements. 

For the past three decades, you could have passed 3.091 [Solid-state Chemistry], never doing a 
problem completely correctly, as long as you earned enough partial credit. Now they have to do 
a minimum of 27 [entire assessments] completely correctly, and most students did many more 
than that completely correctly. I think the average was 33 of the 37. 1 think the problem for 
decades has been to get students to actually practice problems. 

Cima admitted that he cannot be sure that students have actually learned more from the re-designed 
course, despite apparently better performance on the assessments. He does, however, feel that the new 
assessments are a better way to determine what students have learned. His biggest concern about the 
new design is whether students will like it. 

We know that this activity, this term, was much less stressful than the typical course. Virtually all 
the freshman advisors are recording that. It's a question of whether you feel, at the end, whether 
you've learned something. There's this thing at MIT, if it's not hard, it's not worth doing. That's 
systemic at MIT. I gave this talk, an update in my department, about what we've done this term. 
Literally, that's the kind of questions I get: "Shouldn't it be hard?" "Wait a minute, it's freshman 
chemistry, and it's not hard?" That's going to be a big challenge here at MIT. Not just with the 
faculty, but with students who say that, "Gosh, this is so much easier; maybe I'm not learning 
anything?" There's this belief that cramming for an exam, you learn something different than if 
you do it this way, even though these are the same problems. They say, "Well, I can study this for 
this assessment, so it's not like studying for a final, where I don't know what I'm going to get. " 
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Cima reported the following outcomes as a result of the course re-design: 

• The new format of the class appears to have attracted more students. The typical class size of 
320-325 students grew to 361. Despite various glitches during the first two weeks, few students 
switched to the alternative, traditional chemistry class offered on-campus. 

• Part way through the course, Cima surveyed the students to ask if they preferred the weekly 
computer-based assessments or a written exam. Around 80% of the 290 respondents preferred 
the weekly assessments. 

• The number of fifth-week warning flags issued dropped from the typical 30-60 (around 15% of 
the class) to three (less than 1% of the class). Cima believes that getting feedback on 
performance starting the first week, rather than waiting till Week 5, allowed students to adjust 
their study habits immediately to meet the course expectations. 

• Students reported spending eight hours per week on the course compared with 8.3 hours per 
week for the previous year's cohort. 

• Cima believes the new format helped the weakest and least prepared students the most. All but 
two students passed the assessment requirements of the re-designed course. "There is no 
bottom of the class now." However, eight students did not meet the attendance requirements 
for lectures and recitations. 

• Some of the stronger students complained that the new assessment regime denied them the 
opportunity to "shine" because all students were able to do well if they made enough effort. 

• A comparison of student performance in 12 topic areas that were similarly tested in both 2012 
and 2013 indicated that average performance for the class in 2013 was significantly higher on 
every topic. The smallest gain was observed for Conductivity of Semiconductors where around 
15% of the 2012 class obtained full credit compared with just over 50% of the 2013 class. The 
biggest gain was observed for Acids and Bases where only 2% of the class obtained full credit in 
2012 compared with over 95% in 2013. 

If Cima offers the re-designed course again, he may consider including periodic comprehensive 
assessments. While confident that the impact on learning outcomes has been positive, he cautions that 
this process has been labor-intensive and costly, with 33 different people contributing to the course 
design and delivery, including 11 TAs and a team from the Office of Digital Learning to "help with 
software infrastructure." Demand on space and TA time for the assessment room is much higher than 
for a regular class. He estimates that the re-designed course cost two to three times more than a regular 
course. 


Sources: 

Professor Michael J. Cima, MIT; Professor Sanjay Sarma, Director of Digital Learning, MIT; 

Cima, M.J. (2013). My experience teaching 3. 091x. MIT Faculty Newsletter, 26(1), 15-17. Retrieved from: 
http://web.mit.edu/fnl/volume/261/cima.html 

Cima, M.J. (2014). A mastery-based learning and assessment model applied to 3.091r. (Introduction to Solid-State 
Chemistry). Internal MIT Report. 
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Case 7: Re-designing Introduction to Circuit Analysis: 

Khosrow Ghadiri, San Jose State University 

Introduction to Circuit Analysis, a bottleneck electronics and engineering course required of all 
engineering undergraduates at San Jose State University (SJSU), traditionally yields a 65% pass rate. The 
course appeared to be a suitable candidate for a make-over in an effort to improve the passing rate and 
allow students to progress with their degrees in a timely manner. Using edX course materials, course 
instructor Khosrow Ghadiri re-designed the 15-week Introduction to Circuit Analysis into a flipped 
classroom. The re-designed course was offered as EE 098-MIT, one of the three sections of Introduction 
to Circuit Analysis, in Fall 2012. Eighty-six students were registered for Ghadiri's section and were given 
the option to enroll in the new format or in the traditional course offered in another section. None 
switched out of the re-designed section (Ghadiri, Qayoumi, Junn, Hsu, & Sujitparapitaya, 2013). 

Ghadiri et at. (2013) describe the highly structured nature of in-class sessions and out-of-class 
requirements each week. Students in EE 098-MIT were granted full access to the online materials from 
Anant Agarwal's MIT Circuits and Electronics MOOC and also to Agarwal's electronic textbook. They 
were assigned sections of the MOOC lecture videos, embedded questions, auto-graded problem sets, 
and labs to complete outside of class, and were expected to show up in the classroom prepared to 
spend 5-10 minutes answering questions about the assigned material. Ghadiri also asked students to 
rate concepts presented in the MOOC materials in order of difficulty using feedback forms that he 
created. Based on these ratings, he spent 20 minutes of class time explaining concepts that students had 
rated most difficult. Subsequently, the class broke up into groups of three to work collaboratively on 
problems and to complete a group quiz. This activity was followed by a second quiz to be completed 
independently. Sixty in-class assessments were completed during the semester, each designed and 
graded by Ghadiri: a time investment of 80 hours per week (Kolowich, 2013d; K. Ghadiri, personal 
communication, March 10, 2014). Cheal, Associate Vice President and Senior Academic Technology 
Officer at SJSU, reported that while students initially balked at the workload (12 hours per week), they 
were eventually swayed by their own results. 


They were probably doing a lot more work because, previously, if they did the problems for 
homework and they got stuck, they would just stop and wait for class and listen to the lecture 
and try to understand it that way. Now they were working harder, and they were a little 
resentful of that at the beginning. But when they saw how well they did on the mid-term they 
became a lot more enthusiastic. (Cheal, San Jose State University). 

Ninety-one percent of the Fall 2012 EE 098-MIT students passed the course, compared with 59% of 
students in the traditional course offered the previous year. Ghadiri et al. (2013) compared the mid- 
term and final exam results for EE 098-MIT with the results from the previous year's traditional sections 
of Introduction to Circuit Analysis and found that scores were 10 to 12 percentage points higher in the 
re-designed section and that the "lower tail of the curve”(p. 8) had disappeared. Comparing the three 
Fall 2012 sections, they report a mean score on the final exam for EE 098-MIT of 62% (n=78), compared 
with 50% (n=50) and 45% (n= 75) in the two concurrent, traditional sections. The finding that lower 
performers appeared to benefit from the new format is consistent with observations made by Cima in 
his re-designed Introduction to Solid-state Chemistry course at MIT (see Case 6 ). Ghadiri et al. reported 
no significant differences among the three Fall 2012 sections of Introduction to Circuit Analysis in 
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students' starting GPA, suggesting that, on average, students in each section began with similar 
academic abilities. 

At the end of EE 098-MIT, Ghadiri solicited feedback from the students about their course experience 
and asked for suggestions as to how the re-designed course could be further adjusted for Spring 2013. 
While over half the class indicated that access to resources online was what they liked most about the 
new class format, 71% felt that the MOOC material did not correspond well to the SJSU class material, 
and almost half claimed that the online lectures were not helpful, were long, or were difficult. One third 
of the students suggested replacing the online homework problems with sample problems solved step- 
by-step. Ghadiri continued to makes changes to the course and, in Fall 2013, one section of EE 098-MIT 
incorporated hands-on experiments using National Instruments myDAQ hardware and software "...that 
turned student laptops to measurement instruments. Students were able to actually build circuits of real 
components and perform experimentations with myDAQ." (K. Ghadiri, personal communication, March 
10, 2014). 



Photo courtesy of Khosrow Ghadiri 


For Spring 2014, Ghadiri describes further changes to EE 098-MIT : 

Currently. ..SJSU is developing online tutorials, quizzes, midterms, and final exams that can be 
taken on any mobile device such as smart phones, tablets, laptops, and so on. The delivery 
mechanism custom tailors to each student. Students receive material from the topics that need 
improvement based on their past performance. Questions are assigned randomly out of question 
pools with several variables so that the probability that two students have the same question is 
nearly zero. SJSU is developing a security and student identity verification system to protect the 
integrity of online classes. 
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Overall, while the demand on both the instructor's and the students' time has been high, SJSU expects 
that by reducing the number of students who need to repeat Introduction to Circuit Analysis, the 
financial burden on both students and taxpayers will be lower. As the course has been modified each 
semester, it has become increasingly mastery-based and the passing rate in one semester was as high as 
98%. Ghadiri reports that the re-designed section consistently fills to the 90-seat capacity before other 
sections are filled and that, one semester, he offered two sections, both of which filled to capacity. The 
California State University Chancellor's office has negotiated a license with edX to use MIT's Circuits and 
Electronics MOOC across multiple California State University campuses. 


Sources: 

Professor Khosrow Ghadiri, SJSU; Catheryn Cheal, Associate Vice President and Senior Academic Technology Officer, 
SJSU. 

Cheal, C. (2012, August 14). Creating MOOCsfor College Credit (Research Bulletin). Louisville, CO: EDUCAUSE 
Center for Applied Research. Retrieved from http://www.educause.edu/ecar 
Ghadiri, K., Qayoumi, M.H., Junn, £., Hsu, P., & Sujitparapitaya, S. (2013). The transformative potential of blended 
learning using MIT edX's 6.002x online MOOC content combined with student team-based learning in 
class. JUCE (Japanese Universities Association for Computer Education) Journal 2013, No. 3. 

Kolowich, S. (2013d, November 25). A Truce on the Tech Front at San Jose State. The Chronicle of Higher Education. 
Retrieved from: http://chronicle.com/article/A-Truce-Over-Technologv/143229/ 
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Goal 5: Innovation in Teaching and Learning 


Thirty-eight percent of the institutions in our study that were engaging with MOOCs, and nineteen 
percent of our interviewees overall, presented MOOCs as vehicles for experimenting and innovating 
with pedagogy and new models of higher education. Innovation as an end goal was mentioned by 
representatives of seven public universities, three private universities, and one community college. We 
note, however, that the other five goals we report involve innovating as a means to a particular end. 

A number of interviewees portrayed experimentation with MOOCs and online learning more generally 
as preparation for an uncertain future. According to Bacow, President Emeritus of Tufts University, 
"Some institutions are buying an option on the future. If somebody tells you that they know where this is 
going, they are wrong." Harper at the University of Oklahoma exemplifies this position: 

Goal number two is, as an institution, to be forward thinking and adaptable and flexible in a 
really guickly changing landscape that nobody knows where it's going to be in two or three 
years. (Harper, University of Oklahoma). 

Some hold no doubt about the need for innovation in education and that technology will play a key role. 
A senior administrator at an East Coast University noted that the University President "holds the belief 
that residential education is going to change and thinks we can use experiences like MOOCs to change 
the residential experience." Lester at FutureLearn asserts that in order to beat disruption, institutions of 
higher education must get on top of it and can use MOOCs to learn new capabilities and build their 
capacity for moving in new directions: "A digital generation is coming through the pipeline. Literally from 
the age of one kids are using and can interact with technology. Universities have got to be ready." 

Institutions do not want to be left behind in what they perceive could be a game-changing phenomenon 
for higher education: 

We like to be cutting edge and we don't want to see this pass us by and then we are a year 
behind and have to catch up. So we want to make sure we are in the MOOC game and 
experimenting and can see where it goes next, like if it does go down the credit road. (Loble, 
formerly at University of California, Irvine Extension). 

The other goal is somewhat practical: remaining competitive and remaining viable. We don't 
want to position ourselves where what we are doing is a product in a delivery system that people 
do not want anymore. If MOOCs become popular, effective, and a preferred method of delivery, 
we want to be in the game. (Sechrist, Oklahoma City Community College). 

Hathaway at the University of Oklahoma observed that, especially for large state universities, change 
cannot be implemented swiftly so if institutions do not at least experiment with innovative ideas as they 
arise, they will always lag behind smaller, more nimble competitors. 

Frankly, it's quite time-consuming to develop dynamic online content. It's not something that you 
can just say, "Tomorrow, we're going to have it across the broad spectrum that is the University 
of Oklahoma, " or any other university. Other universities are cautioned if they're not getting into 
this phase that, at some point after this experimentation phase, if it proves to be useful to go 
entirely online, you won't be able to just decide one day, "Hey, we're going online" .... Particularly 
for very broad, comprehensive universities like ourselves, seeing and wanting to be a part of the 
possibilities of the future, you've got to start now. (Hathaway, University of Oklahoma). 
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According to Guthrie at ITHAKA, MOOCs have, in the best cases, provided an opportunity for 
administrators and faculty members to work together to experiment with possibilities. As the interest in 
MOOCs and online learning has grown across campuses, several universities have established offices of 
digital learning and have increased their internal capacity for production support. For example, over the 
last two years Stanford University administrators have been engaging faculty in innovation and 
experimentation with MOOCs by providing $25,000 seed grants. Faculty members have been 
encouraged to experiment with a variety of platforms. Similarly, at Cuyahoga Community College (Tri-C), 
the administration pursues a deliberate college-wide strategy to infuse innovative pedagogies and 
technology throughout the curriculum in ways that engage not only their students, but also the faculty. 
According to Miles at Tri-C, the recent creation and delivery of a Tri-C MOOC has contributed 
successfully to this strategy: 

The enthusiasm and the new ideas that have emerged as a result of this initial core team 
working on this MOOC experience has had a multiplier effect in terms of charging other faculty 
or engaging them in new woys.(Miles, Tri-C). 

In other cases, individual faculty members may strive to be at the cutting edge of education and 
routinely experiment with the latest new technologies. Yet others are engaging in MOOCs to help shape 
an innovation that may significantly affect their careers, in preference to sitting by passively. 

Some institutions have specific goals for their innovative efforts in the MOOCspace and note that 
collaboration with partners in this endeavor is key: 

Many of us seek to use MOOCs as an innovation engine where we can experiment with new 
kinds of teaching techniques and student support mechanisms. In collaboration with other 
institutions, the University of Texas System is seeking to develop the learning platform of the 
21st century which will differ fundamentally from a traditional LMS. Working with Stanford, edX, 
and Google, we are helping to drive the development of this new platform. It is less expensive to 
work collaboratively in this consortium than it would be to do this ourselves. And when you have 
Stanford, MIT, and Google involved, you probably have the people who can pull this off. (Mintz, 
University of Texas System). 

Overall it is part of a big grand experiment because no one knows what MOOCs will look like in 
2015-16. ...The goal is being part of that experiment in a purposeful way and engaging in those 
discussions. Not just within Cornell, but with institutions all over the world, not bounded by the 
edX consortium. (Dodds, Cornell University). 

What I like about MOOCs is it gets us involved with other state systems and people who are 
really thinking about the future of higher education in ways that I don't think people would 
otherwise. MOOCs are, from a leadership perspective, a ticket into some really interesting 
discussions that will help us set direction. (Keyek-Franssen, University of Colorado System). 

Schneider at Stanford University's Lytics Lab described a trajectory of innovation in MOOCs starting with 
the initial infatuation with scale, proceeding to a search for a peer-mediated solution to the challenge of 
grading thousands of assignments, to the current recognition that participants may learn best when 
connected with other participants and finding ways to facilitate this through online chat-rooms and 
small group discussions: 

First it was, "Let's make things as big as possible. " That was the driver behind many of these 
courses right from the get-go, and many of the technologies that were being developed to 
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support them, particularly around assessment technologies - both the automatic machine 
grading tools, and also the advent of peer-review and peer assessments. 

Turning the grading over to the students, because you're trying to service many people 
at scale, has meant turning down the authority that lies in the hands of the instructor and the 
TAs, and turning up the authority that lies in the hands of the other students. That has been 
somewhat innovative. 

Chinmay Kulkarni, a doctoral student of Scott Ktemmer's, who is the HCI professor, is 
now working on scaling down again, within this massive setting. Creating little chat rooms or 
other kinds of smaller group-oriented experiences, where the massiveness has this very useful 
aspect to it, but can also lead to a sense of anonymity, or feeling lost. If you then zoom in again, 
and have opportunities for smaller groups to interact with each other, you may be getting 
strength in those ties, and strengthen the whole network and the course overall. (Schneider, 
Stanford University). 

Ironically, while the shift in balance of authority from instructors to students was celebrated as a key 
innovation by the xMOOC players, the idea encountered resistance when the first cMOOCs were offered 
in Canada. George Siemens recounted that he and Stephen Downes, as co-instructors of the first MOOC, 
were initially assailed for diminishing the authority of faculty members in favor of the MOOC 
participants. 

MOOCs were also presented as a disruptive innovation within the higher education business model that 
could help universities become more competitive simply by forcing them to reconsider the status quo 
and to ask fundamental questions about commonly accepted practices in higher education. Dodds at 
Cornell University asserts that, at the very least, MOOCs have pushed college personnel to open up to 
the research about how people learn. 

Not all our interviewees were as convinced that MOOCs would introduce more productive innovation 
than existing experiments with online and blended learning. Interviewees with experience in online 
education and open learning resources over the last two decades pointed out that many MOOCs are 
reinventing and relearning the missteps and successes of online learning and failing to take advantage of 
scale, the most characteristic aspect of MOOCs that differentiates them from other online education. 

We are slightly less persuaded by the notion that there is all sorts of educational innovation to 
do here because while we strongly believe that innovation is important in teaching, lots of our 
staff do online elements in their teaching already. So we are not sure what new ideas MOOCs 
will bring to our students here. (Senior Administrator, Leading UK University). 

lam very pro-innovation, but I'm also pro-take-a-look-at-the-research-before-you-innovate.... 
There are a lot of us in the online learning field who have been basically caught unaware that 
this would catch on so big and, at the same time, they'd be able to completely ignore about 15 
years of research in there. (Vignare, University of Maryland University College). 

It is really just perpetuating the traditional model of teacher-centric learning. We have a quiz 
after 7 minutes and you just listen to someone else lecture, then we have another quiz and then 
we go to the discussion board. How is that innovative? We might as well go to the tele-courses of 
nineties. ...It is not innovative in any way unless you start applying the analytics, unless you start 
leveraging technology. There are so many advantages of teaching online and they are not even 
using any of them. Like the social networking aspect, crowd-sourcing, project-based learning. 
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You have 20,000 people logged in to the same place. Seems like such a waste. (J. Baker, Foothill 
College). 

The value of innovation as an end in itself for educational institutions is hard to measure. As a means to 
achieving other goals it becomes more tangible and we address the evidence of success in achieving 
these in other sections of this report. However, it is abundantly clear that many institutions and faculty 
members have engaged in new educational activities since MOOCs first appeared and that strategies 
employed online such as more frequent assessments and shorter lectures interspersed with questions 
are being taken back on-campus. It is less clear what has been gained by these new initiatives. For the 
most part, institutions are not making any rigorous attempt to assess whether MOOCs are more or less 
effective than traditional on-campus or online courses in terms of improving learning outcomes, and few 
have designed metrics to assess the impact of MOOC-related innovations on other objectives. 
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Case 8: Maximizing the Use of Technology in Learning: University of Oklahoma 

At the University of Oklahoma (OU), open online courses are being developed in collaboration with 
NextThought with a goal of maximizing the power of technology and the Internet. OU courses, offered 
via the Janux Interactive Learning Community, may be taken MOOC-style for free without credit, or for 
credit with normal tuition fees paid. According to Kyle Harper, Senior Vice Provost, "the headline of this 
platform is social learning." Several strategies are being pursued to enhance engagement and learning 
outcomes which Nick Hathaway, Vice President for Administration and Finance, asserts go beyond 
"trying to shove the classroom into the Internet:" 

• Encouraging social learning through allowing participants to interact with one another by 
embedding comments and discussions within the actual content, and controlling with whom 
comments are shared; 

• Game-based learning, applying strategies to improve learning outcomes derived from on- 
campus research; 

• Frequent use of auto-graded self-assessments, the results of which are only available to the 
learner, in addition to assessments that count towards a grade; 

• Archiving courses to allow students to indefinitely re-visit online lectures from a completed 
course, read their own notes or comments, and re-connect with other participants. 

Harper, who taught one of the university's first Janux courses, Law and Justice, was most excited by the 
increased levels of student engagement and participation that he observed as a result of the social 
learning features facilitated by the NextThought platform: 

It's not like we have some really clunky discussion forum under "forums. " We do have forums 
and there are some great debates going on in those, but the social learning is deeply embedded 
in the content. All of the text that I'm using is totally open access text, so we're able to load the 
full text into the platform. Like, Aristotle, who's out of copyright, and the Nicomachean Ethics. I 
can go in there and I can embed, deep in the text, "This is a really important line, " or "What does 
he mean?" I can start a discussion. 

As with other social network systems, there is an activity stream that can notify a participant, for 
example, when another participant adds a comment to his or her note, or engages in a different course 
activity. A student dashboard advertises the current popular discussions within the course and what 
activities are coming up next. The dashboard can be used to indicate who has contributed the most 
posts, drawn the most comments, or garnered the most "likes" for their contributions. It is personalized 
to include each participant's chosen contacts. 

Course videos are accompanied by accurate transcripts which are synchronized such that if the 
participant clicks a spot in the transcript, the video will skip to the same place and vice versa. Comments 
are embedded within the transcript so that a viewer can see relevant notes and discussions from prior 
viewers as the video progresses, or enter comments and questions of their own. All artifacts in the 
course are searchable: video transcript, books, forums, blogs, chats, notes created by the searcher and 
any shared with him or her. 

While the default setting on the platform shares each participant's comments with the whole class, each 
participant can control the setting for every comment and note he or she makes. For example, it can be 
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a private note to self as in a digital notebook, a comment to a study group, a publicly shared 
contribution to a growing discussion on a particular point, or a direct question to the instructor. The 
platform can also differentiate the experience for the MOOC participants vs. the fee-paying participants. 
For example, at the instructor's behest, the fee-paying students can have access to additional e-book 
materials, materials offered under "fair use" that require OU student ID’s, special discussion forums, and 
instructor-graded assignments. 

To further encourage engagement and support interaction, NextThought has developed mobile access 
to Janux courses. Currently, participants can interact with content on an iPad using a browser or by 
downloading a Janux iPad application. Going forward, participants will be able to receive notifications 
and interact with the course content on additional mobile devices including Android tablets and 
smartphones. 

Ken Parker, CEO of NextThought, the company that has built the platform used by Janux, espouses an 
educational philosophy that may well serve to bridge the chasm between xMOOCs and cMOOCs to the 
benefit of learners: 

We want to make it easy and effective for people to connect in an online education experience. 
The Internet transforms by distributing information and connecting people. Current online 
education has primarily focused on the former. Transformation is more likely when we do both. 
The trick is to figure out how to make those connections actually achieve learning objectives so 
that it is not just party time or "here is my latest cat picture." Along with increased engagement, 
we want to help develop higher-order skills such as communication, dialogue and teamwork. 
These are relational activities. It is not enough to simply on-board or import content because all 
you get is the same old stuff in a slightly nicer interface. We must rethink online education in a 
connected world. 


Sources: 

Nick Hathaway, Vice President for Administration and Finance, University of Oklahoma; 
Kyle Harper, Senior Vice Provost, University of Oklahoma; Ken Parker, CEO, NextThought. 
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Goal 6: Research on Teaching and Learning 


Research on teaching and learning was stated as a goal for MOOC initiatives by 28% of the 29 
institutions offering or using MOOCs and by 18% of our 83 interviewees. This was a stated goal for one 
community college, three public research universities, and four private research universities. There is 
considerable overlap between research per se and less formalized efforts to improve teaching and 
learning which many more interviewees alluded to and we reported under Goal 4: Improving 
Educational Outcomes. While often a fine line of distinction, we generally include here work that is 
being conducted by individuals who consider themselves researchers first and foremost, and aim to 
publish their work, while efforts described under Goal 4 were less about creating generalizable 
knowledge and more about improving outcomes in a specific situation. In later sections, we provide 
more detail on Examples of Research being Conducted with MOOCs, Types of Data Available from 
MOOCs, and on Where are we with Data Mining, Learning Analytics, and MOOCs? 

It's all about the creation of new knowledge in a different modality. We can speed up the 
research process because we're going to crowd-source knowledge creation. (Keyek-Franssen, 
University of Colorado System). 

Originally we did the MOOC because we wanted to collect data for educational research. I want 
to have students who aren't saying "Look. I don't want to be a guinea pig. I'm paying $40,000 for 
this education and you're supposed to know what you're doing and be a good educator. " 
(Pritchard, MIT). 

We'd like to help create a collaborative research infrastructure for what we are calling the new 
data science of learning. We can make an important contribution to that because we already 
have a strong integration of computer science into the learning sciences; the Lytics Lab is a great 
example of that, we have computer science students working with education students working 
with communications students all on projects related to this. We are in the process of trying to 
facilitate a conversation between programmers and researchers but we still need to get a handle 
on all the data. (Collier, Stanford University). 

We heard from our interviewees about several types of research being conducted using MOOCs as the 
vehicle for delivering an intervention, or as a source of data on participant behavior and performance. 
The researchers themselves are in a range of disciplines which influence the kinds of questions being 
asked. Computer scientists and engineers have tended to focus either on descriptive studies, 
summarizing demographics or how much each educational resource (video, e-text, discussion forum 
etc.) is used, or on design work creating badging systems, discussion opportunities, peer grading, and 
auto-grading applications. Sociologists and education or "learning science" researchers have more often 
focused on experimental research, for example conducting A/B tests by e-mailing different messages or 
materials to different participants selected at random. 

A great deal of effort is being expended on trying to improve participant engagement and completion of 
the courses and less effort on determining whether participants actually gain skills or knowledge from a 
MOOC. And, beyond improving strategies within a MOOC or applying some of these strategies to on- 
campus and regular online teaching, there has been little research comparing the efficacy of MOOCs to 
regular online or face-to-face courses with respect to learning outcomes. Guthrie at ITHAKA observed 
that this question of efficacy is rarely asked with respect to regular college courses, and doubted that 
there would be any more motivation for such accountability with regard to MOOCs. For now, most 
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instructors and researchers appear content with equating the completion of a MOOC with having 
learned something. But Sarma at MIT asserts that measuring effectiveness will be the next step. "Now 
we think the car works, so next we're going to measure the engine efficiency." How "efficiency" will be 
defined is yet to be determined. 

There are various evaluators and 
research labs in the U.S. and abroad that 
are assessing the impact of MOOCs or 
focusing on MOOC data to conduct 
experiments. Box 6 describes the study 
conducted by the Research and Planning 
Group for California Community Colleges 
( RP Group ) on the San Jose State 
University SJSU+ Augmented Online 
Learning Environment Pilot Project. Case 
9 describes the work of David Pritchard's 
RELATE group at MIT. Other groups that 
have been active in MOOC research 
include the Lytics Lab at Stanford, and the MOOCLab at University of California, Berkeley. The Lytics Lab 
is an informal network of interdisciplinary researchers, including faculty members, doctoral, and post- 
doctoral students who are interested in online learning in general, and in working with MOOC data 
specifically. Disciplines represented include computer science, education, communications, statistics, 
and sociology. 

Reflecting the growing interest in using MOOCs as a vehicle for conducting research, several research 
conferences in 2013 included day-long strands on MOOCs, including the MIT LINC conference in June, 
and the moocshop preceding the Artificial Intelligence in Education conference in August. A December 
2013 conference sponsored by the Bill & Melinda Gates Foundation, the MOOC Research Initiative, 
featured the work of around 30 groups engaged in MOOC research; the 2014 European MOOCs 
Stakeholders Summit included presentations by more than 20 other groups; and the first 
Learning(5)Scale conference in March 2014 showcased over fifty studies or works in progress. 

Areas in which interviewees described research explorations and advances included: 

• The role of social media networks in teaching and learning 

• Testing pedagogical strategies, for example, 

o Firmin, et al. (2013) 
o Champaign et al. (2014) 

• Student engagement and motivation, for example, 

o Coetzee, Fox, Hearst, and Hartmann (2014 ) 
o Kizilcec, Piech and Schneider (2013) 
o Kizilcec, Schneider, Cohen, and McFarland (2014) 

• Machine learning/modeling research, for example, 

o Piech et al. (2013) 
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o Halawa, Greene, and Mitchell (2014) 
o Nguyen, Piech, Huang, and Guibas (2014) 


• Natural language processing 

• Human-computer interaction, for example, 

o Kulkarni et at. (2013) 

o Cambre, Kulkarni, Bernstein, and Klemmer (2014) 

• Personalized/adaptive learning, for example, 

o Buffardi and Edwards (2014) 


• Comparing hybrid with traditional courses, for example, 
o Ithaka S+R (2013) 
o Griffiths (2013) 

o Bruff, Fisher, McEwen, and Smith (2013) 


• Developing data standards and a common platform for data mining, for example, 
o Veeramachaneni, Dernoncourt, Taylor, Pardos, and O'Reilly (2013) 
o Dernoncourt et al. (2013) 


Each topic area is discussed in greater detail in the Sections Examples of Research being Conducted 
with MOOCs, and Where are we with Data Mining, Learning Analytics, and MOOCs . 

Fisher at Vanderbilt University, Parker at NextThought, and others note, however, that the top priority is 
to deliver high quality content online in a reliable manner and that research considerations must be 
secondary. Yet others remain skeptical that MOOC-based research can offer insights that have not 
already been discovered, or that findings from MOOCs can be generalized to other populations and 
contexts. 

There is no need to waste time conducting research on the pedagogy of MOOCs. Existing 
research on online learning has a lot to offer and has already established best practices. 

Research on MOOCs will provide the same insights. (Daniel). 

I cannot believe we are going to spend bunch of time and money rediscovering things we already 
know. Why don't we start with what we know, or what we think we know, design the 
environment based on what we think we know and use the environments to refine what we 
know; let's integrate research and practice. The environments are not being designed to collect 
data in the way that will help us make progress on the big guestions about either course design 
or learning. (Thille, Stanford University). 

The data being gathered are really nothing more than information about these students' 
behaviors. They're clicking here, they're clicking there. They're doing this assignment at the 
eleventh hour. ..but beyond that, it's not telling us what they're learning. It's back to the old 
behavioral scientist kind of stuff. What are we able to observe? And beyond that we can't 
assume anything else. Until we have better assessments on these things, we're not really going 
to be able to draw many conclusions from these data. (Bishop, University System of Maryland). 
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If we were in a consortium with universities that we would not regard as being highly selective, 
how interested are we in the data that comes from the analytics that involve them and their 
potential learners? We are not necessarily. How much is that data going to tell us anything that 
is relevant to what we do? We haven't been convinced yet that we are going to learn anything 
about learner behavior that we need to know and don't know at the moment. We are extremely 
dubious about the expectation that there will be lots to learn from the big data that will improve 
pedagogy both online and on-campus. (Senior Administrator, Leading UK University). 

Jiffy Lube knows more about your car than how much we know about the effect of this 
technology on actual [on-campus] students. They're not the ones who are taking these courses. 
(Fox, University of California, Berkeley). 


Box 6: 

San Jose State University SJSU+ Augmented Online Learning Environment Pilot Project 

One of the most useful applications to date of MOOC platform data to investigate the impact of online 
learning in MOOC-like settings is Firmin, Schiorring, Whitmer, Willett, and Sujitparapitaya's (2013) study 
of the San Jose State University's (2013) Preliminary Summary SJSU+ Augmented Online Learning 
Environment [AOLE1 Pilot Project. This study reviews data from an experiment conducted by San Jose 
State University (SJSU) in partnership with Udacity. In Spring semester 2013 SJSU offered three AOLE 
courses: remedial algebra (Math 6L), introduction to college level algebra (Math 8), and introduction to 
college level statistics (Stat 95). The courses were offered for a $150 fee to a limited number of 
students. In addition to the typical content and activities found in MOOCs, students could obtain online 
help from Udacity Online Support Providers. Participating students were at-risk students, both 
matriculated and non-matriculated, including youth from a nearby high school and students who had 
previously failed a face-to-face version of the remedial algebra course. While prior research (e.g., Figlio, 
Rush, & Yin, 2013; Xu & Jaggars, 2013) indicates that at-risk students are the least likely to thrive in 
online situations, Catheryn Cheal, Associate Vice President and Senior Academic Technology Officer at 
SJSU, explained to us that "...these were the students that we wanted to help the most by the lower cost 
for a MOOC section. It would be a second chance, not normally allowed at SJSU, to pass a previously 
failed remedial math course." 

Both retention and pass rates overall in the AOLE classes were worse than for similar students in 
equivalent face-to-face courses. However, the study found a number of significant relationships 
between variables that might be expected to impact success. Student effort, most notably reflected by 
the number of problem sets submitted by a student, was more important in predicting success in the 
course than demographics or use of student support services. 

Firmin et al.'s (2013) evaluation report provides helpful insights as to what can and cannot be done 
currently with the MOOC platform data and provides a baseline on MOOC data research from which to 
build towards more complex research designs. It also clearly indicated that online educational 
experiences for at-risk students require careful structuring and substantial student support. 
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Obstacles to using MOOCs for Research on Teaching and Learning 

Dillenbourg, Fox, Kirchner, Mitchell, and Wirsing (Eds., 2014) provide a detailed discussion of the 
complexities surrounding the use and sharing of the learner data that is available from MOOC platforms. 
Williams at Stanford University's Lytics Lab identified a number of challenges to realizing the goal of 
using MOOCs for conducting research on teaching and learning, highlighted in Box 7 below. A 
complication expressed by many researchers is that the Institutional Review Boards (IRBs) at different 
institutions interpret and enforce federal human subject research regulations differently. Similarly, each 
institution renders its own interpretation of how to apply FERPA to MOOC-related data. This makes it 
hard to take advantage of the enormous amounts of data potentially available for research purposes. 
Box 8 summarizes the position of Teachers College IRB with respect to MOOC-derived data. Two recent 
“data jams" organized by the White House Office of Science and Technology have facilitated discussions 
at edX and at Stanford on the topic of data standards and sharing, but resolution is not yet in sight. 


Box 7: 

Challenges to Conducting Research on Teaching and Learning Using MOOCs 

Joseph Jay Williams at Stanford's Lytics Lab identified a number of challenges to realizing the goal of 
using MOOCs for conducting research on teaching and learning: 

“1. The biggest obstacle in all of this is that the researchers don't make the decisions about what gets 
taught and how, in a MOOC. Faculty members make the decisions. Udacity owns their MOOC content so 
maybe they can push faculty into having X or Y. But it's not so common. 

2. One big challenge in this kind of research is that, although online environments are fantastically suited 
to doing randomized, controlled trials, it's really pushing people in terms of the kind of statistics they 
need to know, and the kind of modeling they have to do, because issues come up with respect to people 
deciding not to finish your survey, or answering guest ions randomly. Or you have multiple variables 
being related. The same variable you're manipulating might be an influence. There is survivorship 
bias... .What this has meant is that a lot of effects are hard to measure. What we really need in these 
MOOC platforms is more control of the environment. For example, if the surveys are optional, it's hard to 
get conclusive results. 

3. There's no A/B testing framework for content like videos and exercises, or the way something shows 
up. There's none of that in the Coursera or edX platforms as yet. Coursera has introduced A/B testing for 
e-mails. 

4. A big issue is that now we've got all these data, but everyone's collecting data and storing them in 
different ways. Like edX data, Coursera data, Khan Academy data. Everything is in a state where it's hard 
for people to put things together. 

5. Cross-institutional collaborations are extremely complex, and IRBs don't talk well across universities.... 
We need to think very carefully about the regulatory environment because if the government makes it 
easy for people to share data and conduct experiments, it's going to really move forward research on 
MOOCs. If the government puts a lot of things on lockdown and makes it hard to get, for example, 
cross-institutional IRBs, that's going to put a major damper on things like setting up data stream 
collaboratives or making anything open source. (Williams, Stanford University). 
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Issues to consider for future MOOC research 

Several seasoned researchers in the field of educational technology: Barbara Means at the Center for 
Technology in Learning, SRI International; Jessica Heppen, Managing Researcher at American Institutes 
for Research (AIR); and Dede at Harvard suggested that, going forward, it would be useful to track 
course sequences followed by MOOC participants. Both the interest and ability to engage in a higher- 
level course on the same topic could, at least for those participants who do not start with expert 
knowledge, reflect the effectiveness of the initial MOOC. Means at SRI suggested that other longitudinal 
outcomes to follow might, where relevant, include performance on an industry-wide certification exam. 
She cautioned, however, that in online courses the students who drop out tend to be the weakest while 
this is not necessarily the case for face-to-face courses. This creates a problem for comparing the 
treatment-on-the-treated outcomes of the two modalities as there will be a greater likelihood that the 
survivors in the online situation are stronger than those in a comparison face-to-face intervention. 
Heppen noted the importance of only comparing interventions where the performance incentives are 
similar, suggesting that it would not be best practice to compare outcomes from a free, non-credit- 
bearing course with those from a course that costs several thousand dollars in tuition but contributes 
towards a degree. Researchers also raised concerns about attaching high stakes (i.e., fees and credits) to 
experimental interventions with no established record of success. 

Early in 2013, the Computing Research Association sponsored a workshop on New Technology-Based 
Models for Postsecondary Learning: Conceptual Frameworks and Research Agendas . A group of 14 
experts in educational technology from academia and industry, led by Dede from Harvard University and 
Eric Grimson from MIT, gathered at MIT to conceptualize the future role of technology in higher 
education, and to lay out an extensive research agenda. Many of the issues raised in this report as 
priorities for research apply to MOOCs and should be heeded by aspiring researchers in this field (see 
Dede, Ed., 2013). A second workshop sponsored by the Computing Research Association at around the 
same time, Multidisciplinary Research for Online Education, was attended by 94 computer scientists and 
learning scientists. The resulting report, (Fisher and Fox, 2014), similarly lights many paths forward for 
future research on MOOCs and other massive open online education, or "MOOE," including its potential 
impact on workforce development and on enduring interdisciplinary research collaborations. 

Coming from a different perspective, Reynolds from Technology Crossover Ventures questions what the 
appropriate measure of success should be for MOOCs and other online experiences, and raises some 
alternatives to consider: 

It is a hard thing to prove the quality of an education. What is the measure of success, is it 
proficiency in the subject? Is it the ability to get a job, or is it the ability to get a job of a certain 
salary? I think for most people who are attending the online schools they are doing it to drive 
towards some higher station in life. So is that the success vs. actually learning whatever it is you 
are learning in terms of the curriculum? (Reynolds, Technology Crossover Ventures). 

While conducting research on teaching and learning has not been a priority for MOOC initiatives at 
many institutions, there are plenty of researchers eager to take advantage of the opportunities MOOCs 
provide to experiment with a variety of embedded interventions and to work with large data samples. 
While we documented a few examples of such research, it is clear that progress in using MOOCs to 
improve teaching and learning is being impeded by difficulty using the data, which we elaborate on in 
the next two sections, and lack of clarity regarding regulations applicable to the participants and their 
data. 
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Box 8: 

IRB Treatment of MOOC Research: 

What Does and Does Not Require IRB Review? 

According to Teachers College Institutional Review Board (IRB), course-related activities presented in a 
MOOC that result in data that may eventually lead to research do not require IRB approval in advance of 
implementation, or informed consent from the MOOC participants. For example, if all participants are 
required as part of the course activities to complete the same pre- and post-tests of content knowledge 
or pre- and post-surveys (which might address issues including demographics, background experience, 
intentions with respect to course completion, motivation for participation in the course, satisfaction, 
etc.), IRB approval need not be sought in advance of the course. Any such course activities or 
assignments must, however, satisfy the requirements of the instructor's curricular department. 

Before any analysis of such data is conducted with a purpose that extends beyond an internal 
assessment of the course for evaluative purposes and is to be presented as "generalizable research," IRB 
approval must be sought and is likely to be granted under Exemption 4 provided that participants 
cannot be identified. 

Exemption 4: Research involving the collection or study of existing data, documents or records. Sources 
must either be publicly available or information must be recorded by the investigator in such a manner 
that subjects cannot be identified, directly or through identifiers linked to subjects. 

At the other extreme, if some of the MOOC participants will be asked to participate in a research activity 
that is not directly related to the course, or if a third party wishes to collect or analyze data related to 
the MOOC for research purposes, it is clear cut that advance IRB approval would be necessary. 

Where the question of need for advance IRB approval becomes less clear-cut, is when different students 
are offered different activities. For example, if only a subset of the MOOC participants are offered 
participation in online chat-rooms or face-to-face study groups to collaborate on course-related issues, 
or if some, but not all, students are offered extra problem sets or different video and text materials, this 
may be considered more than a purely curricular decision. While such strategies may in some cases 
constitute routine pedagogical approaches, if the variations are planned with the intent to conduct 
research, the Teachers College IRB office advises researchers to check with their own IRB offices 
whether an advance review is necessary. 

Source: Teachers College IRB Office. 
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Case 9: RELATE at MIT 

David Pritchard's RELATE (Research in Learning and Assessing and Tutoring Effectively) education group 
at MIT has been analyzing data from Pritchard's Mechanics Review MOOC to capture the impact of this 
learning experience on educational outcomes and to improve successive iterations of the course, with 
the aim of building participants' expertise in problem-solving. The course is designed using a structured 
pedagogical strategy, "Modeling Applied to Problem Solving," developed by Pawl, Barrantes, and 
Pritchard (2009). 

Mechanics Review was originally created at MIT in 2009 as a three-week "flipped" review session on 
Newtonian mechanics for on-campus students. It was subsequently developed into a full, semester-long 
course with more online preparation. Since 2012, the resulting online course has been offered three 
times as a MOOC, first through the LON-CAPA platform and more recently via edX. From the beginning, 
the course was set up as an experiment to investigate how to improve students' conceptual overview of 
mechanics, as well as their ability to use "expert-like" approaches to problem-solving. For RELATE, the 
appeal of offering a MOOC was the promise of large datasets: 

...we have made an accelerator for physics education research... it's like a particle accelerator in 
physics. You have a high intensity stream: in this case, students. You can put something in there 
for them to interact with. Then you have a whole host of data mining and machine learning and 
psychometric tools available. ..that's like the detector. ..to study what comes out of this. 

...if you want to do good research, you really need 1,000 people. Four hundred is not really 
enough. Once you split them up, you've only got 200 people per group, then you just don't have 
enough statistics to detect a fifth of a learning effect size. (Pritchard, MIT). 

A team of post-doctoral fellows with complementary skills in data mining, psychometrics, and education 
research has used Item Response Theory (Hambleton, Swaminathan, & Rogers, 1991) to measure the 
week-by-week skills of Mechanics Review participants and to study the correlation between time spent 
on various activities with improvement in problem-solving skills. Champaign et al.'s (2014) paper 
"Correlating Skill and Improvement in 2 MOOCs with a Student's Time on Task” presents a detailed 
description of the data analysis procedures and identifies a number of challenges arising in MOOC data 
analysis. The researchers also administered pre- and post-tests to measure absolute learning in the 
MOOC and have compared this with learning in traditional, on-campus, lecture-based courses. 
Recognizing that the populations online and on-campus are quite different, RELATE also measures the 
increase in skills for MOOC vs. on-campus participants. 

Using the participant activity data ("clickstream data") that is available from edX, RELATE has been able 
to accurately operationalize concepts such as "time spent reading the book" (after allowing for instances 
where the student leaves his or her computer without logging out), and "looking at a book page" (which 
excludes flipping through a number of pages). This has allowed the team to identify patterns in the data 
such as markedly different allocation of time and use of various resources by MOOC participants when 
working on homework as opposed to exam problems. This seemingly straightforward analysis requires 
significant programming skills and investments of time, which few other institutions have been able to 
muster. 

Findings that have been used to modify the course over the successive iterations include the 
observation that frequent checkpoint questions and weekly quizzes increase the likelihood that 
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participants will read the course materials and complete the course. Additionally, a flexible schedule 
with all relevant materials available at least a month ahead of deadlines can boost retention rates. 
Online support, according to Pritchard, has evolved from none to "wiki 1, wiki 2, then an e-text, then an 
e-text with homework, then an e-text with homework with different graded levels." 

Pritchard observed that extreme diversity of incoming knowledge and skill levels among MOOC 
participants presents an obstacle in attempts to isolate the impact of the MOOC, or of specific 
components, on skill improvement. For example, while one might expect that participants who made 
greater use of the course materials would show a higher skill level, in MIT's 6.002x Circuits and 
Electronics MOOC that Pritchard also studied, the highest performers were actually the ones who spent 
least time on homework and class materials. In Mechanics Review, 25% of the participants were physics 
teachers while another 25% had a high school degree or less. Many of the high performing participants 
started out already proficient in the course concepts and could do the homework and tests quickly and 
accurately without having to learn the material first. However, more time spent on the checkpoint 
questions (questions embedded in the e-text that tested conceptual knowledge) in Mechanics Review 
did correlate with greater skill improvement (see Champaign et al., 2014). 

RELATE plans to use A/B testing, which they expect to be available on the edX platform by Summer 
2014, in future iterations of Mechanics Review. Possible tests will include varying the order of problems 
in assignments or using different instructional materials to continue fine-tuning the delivery of the 
course in order to maximize student performance. Ultimately, the goal is to build sufficient trajectories 
of how different types of students learn best to allow creation of a real-time adaptive learning 
environment in which the content, problems, and support presented to each student vary depending on 
how (s)he progresses through the course activities. 


Sources: 

Dave Pritchard, Cecil and Ida Greene Professor of Physics, MIT 

Borkovitz, D. (2013, July 30). Summer MOOCs 2: Mechanics Review. [Web log post]. Retrieved from: 
http://debraborkovitz.com/2013/07/summer-moocs-2-physics-review/ 

Champaign, J., Fredericks, C., Colvin, K., Seaton, D., Liu, A. & Pritchard, D. (2014, March). Correlating skill and 

improvement in 2 MOOCs with a student's time on task. Paper presented at Learning@Scaie Conference, 
Atlanta, GA. Retrieved from: http://dx.doi.org/10.1145/2556325.2566250 
Carson, S. (2013, May 1). An MIT physics MOOC five years in the making. MIT News. Retrieved from: 
http://web.mit.edu/newsoffice/2013/mechanics-review-mitx-course.html 
Scarsonmsm. (2013, May 27). Mechanics Review MOOC: Learning about digital learning through data [Web log 
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Examples of Research being Conducted with MOOCs 


In this section we provide further details regarding each of the types of research being conducted with 
MOOCs, as listed in the section Goal 6: Research on Teaching and Learning. 

The role of social media networks in teaching and learning 

Siemens described a recently-begun 5-year project with Caroline Haythornthwaite of the University of 
British Columbia and Anatoliy Gruzd of Dalhousie University, funded by the Social Sciences and 
Humanities Research Council of Canada, to investigate the development of digital literacy and creativity 
in emerging technologies using participatory approaches. The overarching research question is "How do 
social media networks influence educational models?" One goal is to devise "theoretical and technical 
solutions that will help educators and researchers to study learning processes in social media" and 
another is "to provide educators and learners with tools that will help them to determine whether their 
particular use of social media is beneficial to their teaching or learning." (Dalhousie University, June 4, 
2013 announcement ). 

Testing pedagogical strategies 

Several interviewees described MOOC initiatives as an opportunity to explore how to improve current 
teaching practice. 

Even if the hype for MOOCs dies out in five years, things would be different. The game has 
changed because of them. People are rediscovering how people actually learn: what you need to 
do in order for people to learn. These experiments were happily ignored for years. (Dodds, 

Cornell University). 

Research really is a supporting mechanism for improving professional development. Learning 
more about what works, what doesn't work. Looking into different design models and ways of 
implementation. (Kleiman, North Carolina State University). 

Kleiman's group in particular is less concerned that participants complete the MOOCs they are offering 
and more concerned that the courses are designed to allow participants to get what they need from the 
experience, whether that involves completing only two specific units, or the entire sequence. Pre- 
surveys elicit details on the participants' goals, and mid-course surveys inquire as to whether 
participants' goals are being met. By tracking resource use and activity, Kleiman is developing an 
"informal taxonomy of ways different participants get through a course" and re-designing the MOOCs in 
an effort to meet the participants' stated needs. Lester at FutureLearn similarly stresses that the point of 
making better learning experiences should be about meeting the objectives that the learner sets for him 
or herself, as opposed to those set by an external entity. He suggests that MOOC designers consider 
applying Argyris' (1978) concept of single vs. double loop learning, whereby they do not simply try to 
continuously improve the learning experience as it stands, but question the entire approach and 
whether they are even teaching what the learner wishes to learn. 

One particular mechanism that several interviewees expressed an interest in using for the purposes of 
discovering what works best in teaching is A/B testing, but the technical capability for this is still fairly 
limited. Coursera allows for randomly selected MOOC participants to be e-mailed different messages, 
but cannot yet offer alternative versions of resources within a course. Pritchard from MIT reported that 
A/B testing was possible with an earlier version of the edX platform but that the feature was sidelined in 
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later versions. However, he and others are collaborating with edX to re-instate the feature by Spring or 
Summer 2014. 

There has been limited use of pre- and post-tests in MOOCs as a means to establish what knowledge 
and skills participants gain as a result of participation in the courses and how various pedagogical 
strategies impact these outcomes. Williams at Stanford University reports that Jane Manning, Director 
of Platforms, Office of the Vice Provost for Online Learning at Stanford, has been working with a few 
Stanford faculty members to pilot pre- and post-test items within MOOCs. Case 9 describes work 
conducted by Pritchard's MIT-based RELATE group that includes the use of pre- and post-tests in a 
physics MOOC, Mechanics Review, to assess improvement in problem-solving skills as a result of 
engagement in the MOOC activities. We look forward to publication of findings. 

In reality, the most useful data collected so far that can be practically applied towards improving the 
online learning experience, appears to have come from surveys of the participants during or after the 
MOOC. Firmin et al.'s (2013) study of the San Jose State University's Augmented Online Learning 
Environment (AOLE) pilot (see Box 6) included three surveys of the students that surfaced substantial 
feedback, albeit from a relatively small number of students. For example, students suggested providing 
more up-front information about availability of support services and how to access them, consolidating 
all course activity on a single website, clarifying course requirements, and organizing forums by course 
and topic. These suggestions could be swiftly adopted for the next set of courses offered. 

Fox at University of California, Berkeley suggests that the availability of substantial amounts of data 
from MOOCs will prompt instructors into "being more scientific in what we do" with regards to 
improving courses and pedagogical strategies from one semester to the next. For example, it will 
become possible to apply item response theory to test the quality of the questions presented in 
assignments and exams. Fox also described an investigation being pursued at University of California, 
Berkeley with regard to the importance of a specific faculty member's presence in a MOOC. 

They're actually looking to do an experiment to tease apart how much of the magnetism of a 
MOOC is inherent in the materials versus how much of it relies on the presence of that specific 
instructor. If I take the exact same materials, but a different person facilitates the course, can I 
measure the effect that has on retention, and student performance, and so on? (Fox, University 
of California, Berkeley). 

Student engagement and motivation 

A number of experiments have been conducted to investigate how to motivate students in online 
environments, primarily to promote active participation and persistence. However, as Heppen at AIR 
points out, it is important, but difficult, to establish incoming or baseline motivation before trying to 
assess whether it is "malleable." 

The MOOCLab at University of California, Berkeley investigated the impact of social reputation features 
on retention and performance in a software engineering MOOC, Software as a Service. The researchers 
replaced the standard edX forum with two variants, a basic forum and one in which participants could 
"upvote and downvote" (Coetzee et al., 2014, p.1178) comments, questions, or answers in the forum, 
earn a reputation score based on the usefulness of their posts, and eventually gain access to moderation 
features in the forum. Participants in the MOOC were assigned at random to either one forum or the 
other on their first visit to the forum, and could not participate in the other forum. Coetzee et al. found 
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that the social reputation features encouraged quick response times to forum posts and more responses 
per post but did not improve grades, retention, or participants' reported sense of community. 

Taking advantage of learning analytics techniques, Kizilcec et al. (2013 ) investigated levels of 
engagement in MOOCs using participant activity data in three computer science MOOCs. They discover 
four different categories of learners in MOOCs, reflecting diminishing levels of engagement: completing, 
auditing, disengaging, and sampling learners. For example, auditing learners stay engaged throughout 
the majority of the course, but do not complete assessments. These differences in approach to MOOC 
participation underscore the variety of goals participants have for enrolling in the courses and present 
an opportunity to personalize the course experience to better meet these varying needs. Kizilcec et al. 
also show associations between multiple activity types, for example, that participants who are more 
active in discussion forums are also more likely to complete a MOOC. However, it is not clear that there 
is any causal relationship. 

Using the A/B testing feature offered by Coursera, Kizilcec et al. (2014) sent out different versions of e- 
mails to MOOC participants to assess the most effective strategies for encouraging engagement in the 
discussion forums. E-mail messages included neutral reminders to all participants to use the forums and, 
later in the course, more targeted messages to participants who had not used the forums. Targeted 
participants received one of three messages: one implying that forum participation yields collective 
benefits, one implying benefits to the individual, or a purely neutral reminder that the forum existed. 

The researchers concluded that the encouragements did not have the desired effect, and, to the 
contrary, the "collectivist" messages were associated with less participation in the forums than if no 
messages were sent at all. 

Rene Kizilcec, Daniel Greene, and Rachel Baker have been exploring "implementation intention" by 
assigning, at random, different versions of a pre-course survey to the participants in a MOOC. The 
survey includes a question about intentions, for example, "How many videos will you watch per week?" 
For some participants an additional question asks "What is your plan for making sure you meet this 
goal." Impact on completion and performance is being investigated. To control for engagement, other 
participants are simply asked how many videos they think the average participant watches. 

Daniel Greene at Stanford's Lytics Lab is investigating the impact of Carol Dweck's (2008) concept of 
"growth mindset" on dropout and completion in a MOOC context by providing some participants with 
information on the idea that intelligence is not fixed and can be improved through effort, and asking 
them to reflect on a situation in which they worked hard and improved in what they were doing. 
Williams, also at Stanford's Lytics Lab, has been investigating the impact of introducing pop-up messages 
to MOOC participants while they are engaged in problem-solving. The goal is to simulate a tutor asking 
questions about the student's progress and strategy in order to prompt self-monitoring and to suggest 
study strategies. 

Miles at Tri-C reports using data from the college's Pre-algebra CCC MOOC to understand student 
behaviors such as persistence and time on task, with a view to developing competency-based learning 
experiences for Tri-C students. 

Machine learning/modeling research 

At Stanford's Lytics Lab, Sherif Halawa, working in conjunction with MIT's Computer Science and 
Artificial Intelligence Laboratory (CSAIL), has been building a model to predict dropping out in MOOCs 
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based on video-watching behavior (see Halawa et al., 2014) . According to Williams, Halawa intends to 
add an experimental component in which students who may be at risk of dropping out will be sent an 
intervention message. 

Also at Stanford, Jonathan Huang, a post-doctoral fellow in computer science, and Chris Piech, a 
graduate student in computer science, are engaged in the project Code Webs to facilitate the 
categorization and automatic grading at scale of students' programming assignments (see Nguyen eta!., 
2014) . Huang and colleagues are also investigating how to use machine-learning to improve peer 
grading, for example, how to direct assignments to students for grading, and how to weight the peer 
grade a student gives by using information on how reliable the grader is, or how consistent (s)he is with 
other graders. Piech et al. (2013) analyzed peer grading activity in two consecutive offerings of Scott 
Klemmer's Human Computer Interaction MOOC and found, among other things, that "students are least 
biased when grading peers with similar score." (p. 6, para. 8) 

Natural language processing 

A number of interviewees commented on the need to find ways to better manage and navigate MOOC 
discussion forums given how quickly they become difficult to follow due to the sheer volume of 
commentary. Fisher suggests that this will be an area of research interest for some Vanderbilt faculty 
members. 

We have strong natural language processing expertise. How do we use a natural language 
processor, for example, to grapple with complexities of discussion boards at a global level? Right 
now the equipment is there. The hardware is sufficient. Through Coursera we have discussion 
boards that have many thousands of users, many thousands of simultaneous entries. Response 
time is quite good. But for someone who enters the course three days after it starts, it is already 
impossible to navigate the discussions. How can we use natural language processing to 
reorganize those discussions over time? That would be area of research our Vanderbilt faculty 
would want to get into. (Fisher, Vanderbilt University). 

Human-computer interaction 

Interviewees at both Vanderbilt University and Stanford University described the use of MOOCs in 
current or planned work on human-computer interactions. For example, Chinmay Kulkarni at Stanford 
University, working with Scott Klemmer (now at University of California, San Diego) created a 
functionality that allows students to grade each other's work based on a rubric, and has conducted 
analyses of what makes for good peer grading (see Kulkarni et al., 2013 ). Kulkarni recently launched 
talkabout, an application that plugs into Coursera allowing MOOC participants to engage in video 
discussion with a small group (five to fifteen) of peers through Google Hangouts (see Cambre et al., 
2014) . Kulkarni is also investigating what makes discussions work well, for example, the optimal gender 
composition of a group. 

Ragan at Penn State World Campus reported that educational psychologists at the university are 
investigating how to create a "graduated peer assessment mechanism, how to validate those people 
who are good assessors, and have them move up in status so that you can really tap into the expertise 
within your community." 
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Personalized/adaptive learning 


Currently, the Holy Grail in education circles appears to be personalization, at scale, of learning through 
the adaptiveness of the learning experience. The scale requirement is most likely to be met through 
online delivery, and the massive amounts of data that are available from courses like MOOCs improve 
the possibility of creating such adaptive learning experiences. As a professor from an East Coast 
University points out, "These massive datasets might actually allow machine learning to uncover 
patterns that no human can actually see, precisely built on the model of Watson." 

Dr. John Whitmer, Program Manager, Academic Technology & Analytics Office of the Chancellor, 
California State University, emphasizes that although the potential exists for platform-driven just-in-time 
feedback to students, as evident in applications such as Course Signals, Blackboard Analytics for Learn, 
and Blackboard Retention Center, the necessary algorithms have not yet been incorporated into 
standard learning management system applications or MOOC platforms. 

Currently, beyond the large publishers who can afford the development of adaptive learning systems, it 
appears that academic research and development of adaptive learning is limited to specific applications, 
primarily in the field of computer science. For example, at University of California, Berkeley's MOOCLab, 
Armendariz, MacHardy, and Garcia are developing OCTAL (Online Course Tool for Adaptive Learning), 

"an adaptive exercise system that customizes the progression of question topics to each student" (see 
MOOCLab Publications page) for use in computer science courses. At Virginia Tech, Buffardi and 
Edwards have created CodeWorkout, "an online learning environment that offers drill-and-practice 
exercises with novel social and adaptive scaffolding" ( Buffardi & Edwards, 2014, p. 724 ) to help 
computer science students practice programming. 

In order to provide adaptive learning that is more than superficial presentation of alternative problems 
or pathways, it must be possible to follow the trajectory of any one student through the course 
materials. Currently, the platform data from the major MOOC platform providers cannot present 
individual student trajectories in any easily usable manner. 

Comparing hybrid with traditional courses 

Numerous instructors have been experimenting informally with the incorporation of MOOCs or MOOC 
material into their face-to-face courses and have compared outcomes of the blended/hybrid versions of 
the course with those reported in the former traditional format. We document several of these 
instances under Goal 4: Improving Educational Outcomes . Griffiths at Ithaka S+R has been leading a 
comprehensive research study to investigate the impact of embedding customized MOOCs and other 
online learning experiences into existing courses at campuses across the University System of Maryland 
( Ithaka S+R, 2013; Griffiths, 2013) . The goal of these initiatives is to improve educational outcomes for 
students and to lower costs. Of the 22 tests conducted in 2013, five were side-by-side comparisons in 
which a MOOC was embedded in one section of a face-to-face course. Outcomes are being compared 
with simultaneously-offered traditional sections. In addition to comparing completion rates in the 
courses, common assessments were used across the two sections to allow for a comparison of 
performance. While ideally instructors and students would have been assigned to sections at random, 
this was not practically feasible, a challenge often encountered by researchers in on-campus higher 
education settings (e.g., Bowen, Chingos, Lack, & Nygren, 2012; Figlio, Rush, & Yin, 2013). In addition to 
the side-by-side comparisons, ten case studies were conducted of courses in which the instructor 
selected one or more MOOCs for blending with their own course materials. 
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While results of the tests are pending, several implementation challenges are noted in the interim 
report (see Ithaka S+R, 2013). Faculty members adopting MOOCs created by others usually wish to 
customize the MOOC to fit optimally within their existing curricula, often requiring that the various 
components are unbundled. While technically possible, creators of MOOCs are not always receptive to 
this request and sometimes withhold permission for their courses to be adapted in this fashion. 
Intellectual property rights issues remain unresolved, resulting in uncertainty regarding ownership of 
the adapted course and ongoing access to externally developed MOOC materials. Complications arise in 
attempting to license materials that include copyrighted content. MOOC platforms do not integrate 
easily into existing learning management systems, requiring instructors to choose one platform or other 
as the primary interface for their course, and information technology departments to establish links 
between the systems. 

At Vanderbilt University, Bruff et al.( 2013) investigated students' perceptions of a blended course in 
which Stanford University's machine-learning MOOC was "wrapped" with content provided by a 
Vanderbilt University instructor (Fisher) and delivered on-campus. Given student concerns about the 
alignment between the MOOC content and in-class components of the course, Bruff et at. suggest 
incorporating only parts of a MOOC, or parts of several MOOCs, to better integrate the online and in- 
class elements. Both Ithaka S+R (2013) and Bruff et al. suggest that MOOC creators consider building 
courses to allow for the possibility of deconstruction by adopters. 

Note: The item "Developing data standards and a common platform for data mining" is addressed in the 
section Where are we with Data Mining, Learning Analytics, and MOOCs? 
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Types of Data Available from MOOCs 


This section provides a detailed description of the data that is available from MOOC platforms and the 
subsequent section on Where are we with Data Mining, Learning Analytics, and MOOCs? addresses the 
potential for and obstacles constraining use of these data to improve teaching and learning. 

Interviewees described three main sources of data from MOOCs. One is the "clickstream" logs collected 
directly from the course platform through server registration of user activity: what Heppen at AIR 
described as the “data exhaust" produced by any type of learning management system (see Box 9 ). The 
raw clickstream data is in the form of log files which record each and every time any platform user 
accesses a resource from the server, for example, watches a video, stops a video, or enters quiz 
responses. According to Koh at Coursera, anonymized event logs are given to an appointed data 
coordinator at each partner institution who is responsible for appropriate Institutional Review Board 
(IRB) clearances for researchers to use the data: 

We have certain mechanisms in place to anonymize the data so we strip out the student name 
and e-mail address and instead use unique numeric identifiers so that researchers can do their 
research without having to get IRB waivers and don't know any more than they need to know 
about the students' private information. (Koh, Cousera). 

Programming scripts are required to pull out useful information such as the number of individuals 
submitting quiz responses, or length of time each video is watched. This is the data that is used for data 
mining and learning analytics. The report section on Where are we with Data Mining, Learning 
Analytics, and MOOCs describes in detail the challenges in using these data but, without exception, 
interviewees who had tried to engage with the platform data asserted that they are difficult to use. 

Right now, it is so hard to get the data into a way that makes sense so we can do the research 
that we need to do to look at effectiveness. (Collier, Stanford). 

Infrastructure and platform decisions are made with the user in mind, but they're not made with 
research in mind. They can create real complications in understanding, and being able to do the 
interesting work of tracking how people navigate the course. (Heppen, AIR). 

Rizvi at Pearson, who is familiar with many learning management systems, notes that these issues are 
not endemic to MOOCs, but arise with all educational technology products because the "products were 
led by technology as opposed to pedagogy, and were not designed to ask the right questions." 

A second data source is the summary tables made available by the platform providers. At least one of 
the platform providers, Coursera, provides an administrator dashboard, essentially a bank of tables, 
summarizing some of the most commonly requested data points. This allows the instructor to see how 
many participants are active in the course, how many complete each quiz, how many attempts are 
made, how many answered a question correctly, and so on. These data are reported in the aggregate, 
not by individual student, and does not show relationships among variables, for example, whether 
participants who watch more videos perform better on quizzes. To conduct such investigations requires 
additional programming efforts to parse raw data. According to Koh at Coursera, the demand for the 
clickstream data is high and growing: in the first four months of 2014 Coursera exported data from 776 
sessions. 

I think we receive an average of one request a day from schools for raw data exports for their 
own MOOCs, which we provide as part of our agreement with them. They are not usually asking 
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for data from other universities, although that is growing, but most people are studying their 
own classes. But we do have researchers who have been reaching out across institutions and I 
think that's actually one of the most promising aspects of this, that if it's on a common platform 
and a common data format then it's easier to access everyone else's data provided that people 
opt in and understand that they are sharing the data. 

We are slowly writing more and more scripts and adding them to the site but, in the 
meantime, if you want to do anything else with the raw data, and there is a lot of stuff that 
people want to do, then they can go ahead and do that, and then we encourage them to share 
the scripts with each other and make them public. Also, you can use programming languages to 
write your own assessments and then there are ways for the administrators to plug those back 
into our interface. They can do this without talking to us. (Koh, Coursera). 


EdX also provides raw, but not anonymized, data 
to a data coordinator at each partner institution 
and offers detailed guidance on how to work 
with the data files (see edX Data 
Documentation) . According to Petersen, the edX 
learning analytics team is currently working 
towards the creation of data dashboards, but 
the responsibility for writing scripts to parse the 
data rests with the consortium schools. Petersen 
notes that, being an open platform, edX has a 
repository on GitHub and consortium members 
are encouraged to share their scripts. Led by 
researchers at Stanford University and MIT, 
there are promising conversations with edX, 
Coursera, Udacity, and others towards an 
interoperability framework, called MOOCdb, for 
data to facilitate cross-platform analyses (see 
Veeramachaneni et at., 2013), but, according to 
Petersen, it will take some "pushing" from 
members of multiple platforms to catalyze this 
process. 


Thirdly, most institutions offering MOOCs create their own pre- and post-surveys which are 
administered via the MOOC platform, and are usually optional for participants to complete. Pre-surveys 
investigate items such as participant demographics, pre-existing familiarity with the course content, 
other MOOCs completed or in process, how participants learned about the MOOC, reasons and goals for 
participation, and prior online learning experience. Fisher at Vanderbilt University pointed out that U.S.- 
centric surveys, demographic questions in particular, may not be understood by cultures across the 
world. Post-surveys typically include questions to elicit feedback on the course design such as 
accessibility and usefulness of the various types of materials and activities, and how much time was 
spent on the course. Additionally, participants are often asked whether they achieved their goals, how 
satisfied they were with the experience, and how they might apply what they gained from the course 
going forward. Some institutions also include a mid-term survey to obtain feedback on the student 
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What are the Platform Data? 

"All the course specific data are in one set of 
databases, which are relational or SQL databases, 
which have a series of tables. This includes the user 
behavior within the course like the quiz data, forum 
posts, and so on. The demographic data are not 
linked to a course: we collect this on the platform 
level because one student might take multiple 
courses but we only need one set of demographic 
data. This is also a relational SQL database, that is, 
it has a set of tables. So we export both databases 
separately and we provide a way to link them up 
because they use the same common identifiers for 
each participant. A third database is the clickstream 
data, which is just basically one long event log or list 
that shows that, at this time, someone did this, and 
it just keeps getting longer throughout the course. 
You have to write scripts to parse that data." 

Source: Pang Wei Koh, Coursera. 
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experience and course design. Sample surveys are available from the American Museum of Natural 
History and Penn State University in Appendix V . Because these surveys are usually optional, the 
response rate is typically low (25% - 40% according to a few of our interviewees) and not representative 
of the entire population. The post-surveys suffer from "survivorship" bias. One institution that made 
pre-surveys mandatory, Cuyahoga Community College, found that 59% of the course enrollees were 
willing to complete the survey in order to access the course materials. The completion rate for this 
MOOC was higher than for most MOOCs: 18% of actual participants, or 11% of initial enrollees versus 
the more typical 5% observed. Perhaps the most non-committal students were weeded out by the 
survey requirement. 
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Where are we with Data Mining, Learning Analytics, and MOOCs? 


While there is no doubt that the MOOC platforms collect oceans of data, it is apparent that interpreting 
these data and querying them to answer useful research questions is a work in progress. As Newfield at 
the University of California, Santa Barbara succinctly states: "It's a strange field where the promises and 
the commercially-driven claims are way out in front of the actual data." More neutrally, Petersen at edX 
observes: "...as much as we have been talking about learning analytics for years and researching big 
data, it is still something that is emerging. It is in its forming phase..." For all the volume of data, 
Petersen cautions that unless analysts start with a good educational research question, there may be 
little to be gained. Ryan Baker, President of the International Education Data Mining Society (IEDMS), 
steering committee member of the Society for Learning Analytics Research (SOLAR), and instructor for 
the MOOC Big Data in Education, similarly recounts that educational data mining and learning analytics 
is "an incredibly newfield: there is no standard text, and everyone has to learn through apprenticeship." 
He notes that while computer-assisted instruction has been around since the 1960s, and educational 
software since the 1980s, "decent log files" have only been available since around 2006. Baker is 
concerned that many computer scientists are attempting research using MOOC platform data without 
attending to what has already been learned in education research over decades, and in data mining and 
learning analytics over the past few years. 

Whitmer at California State University describes a major gap between promise and reality with respect 
to research using MOOC data. As a member of the research group studying the SJSU+ AOLE experiment 
with Udacity, funded by the National Science Foundation, (see Box 6 ) and also Mt. San Jacinto 
Community College's Crafting an Effective Writer MOOC, funded by the Bill & Melinda Gates Foundation 
and delivered via Coursera, he has grappled with data from two of the major MOOC platform providers. 
Some of the problems are relatively easy to fix. For example, one platform did not initially store any data 
on the discussion forums, apparently believing them to be unimportant, while another did not record 
student views of static content pages, a problem when reading online text is an important component of 
the course. A more major hurdle Whitmer identifies, common to all online platforms, is that data logging 
systems were established to record the activity occurring on platform servers. They were not designed 
specifically to track how learners act, either individually or in aggregate. He alludes to the current data 
situation as being "almost chaotic when you get under the hood" and suggests that the MOOC platform 
providers are understating the gap and generally lack educational research expertise. Trying to use the 
platform data to answer research questions is currently, according to Whitmer, like “trying to build a 
spaceship at the atomic level." While it is possible to tease out aggregate data on participant activities in 
MOOCs, more useful inquiries such as looking for patterns in learner behavior over time and over 
multiple courses, or relationships between activity and measures of achievement, are not routinely 
feasible. 


The MOOC platform providers are saying you can improve pedagogy and learn from research 
using MOOC data. The data may all be recorded, but getting something meaningful out of them 
is a very large, resource-intensive job, and most people don't have enough time or resources. The 
person I listen to the most is Daphne Koiierfrom Coursera say how much hope there is for what 
we can learn from the data in MOOCs. Every time I hear her say that, I think, "What a great idea 
and vision. We are really far away from it." The MOOC platform providers are leading people to 
believe this is possible, but they're not putting the time and resources in to make it happen. 

I don't think they're the right people to do it. They don't know the questions to ask. They don't 
know how we analyze data. They really don't know what rigorous statistical analysis is if you 
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look at any of the results of the stuff that they do. Like when they say what their population is, 
they don't even give a sample size. They don't say what percentage of people respond. I think to 
do this well, they need to look outside of their own shop and consult or partner with the 
educational research community, like the whole SOLAR community, Ryan [Bakerj's work, George 
[Siemensj's work, Simon Buckingham's work. A lot of us have already done this stuff and 
understand the domains, both the online teaching and learning domain, as well as the research 
domain. There's just a lot of expertise that could really be brought to bear on this to make it go a 
lot further and a lot faster. (Whitmer, California State University). 

Fox from University of California, Berkeley described the difficulty involved in answering apparently 
straightforward questions with the data from the edX platform, and Pritchard from MIT provides a 
graphic account of the challenges in parsing the platform data for a particular MOOC: 

There's one data store that holds the registry of students. To the extent demographic 
information is available, it's available in that database, but that's separate from the one where 
marks are recorded for individual assignments. Then there's a separate database which 
essentially records what we can think of as the clickstream, the student interaction with the LMS. 
If you want to ask the question of looking at students from the Americas as a group, "Are they 
doing better or worse on these homeworks, and does it have any correlation with how much 
they're interacting with the segments of course material that cover those topics?" That's three 
different databases that you have to cross-correlate and join across just to do that relatively 
simple query. (Fox, University of California, Berkeley). 

We probably invested nine computer science post-doc months in dealing with the data from 
Anant Agarwal's Circuits and Electronics MOOC. The data comes off a cloud, or 32 servers that 
have logs. I think we had 10,000 individual logs. When we put them all together, we had 110 
gigabytes of data, which is like a small-town library. It's like 100,000 books. This is for the 7,000 
people who got certificates. It's not an amount of data that you can put on your desk. 

If you try to answer a straightforward question like, "How many pages of the book did 
the student look at?" the log just says that the person went to page 312 and then, over the last 
half hour, has clicked up pages eleven times and back pages four times. So you have to figure out 
that they started on page 312 plus eleven minus four, to figure out what page they're on. Then 
you have to decide whether they looked at it. We have to look at the plot of how long people 
looked at each course, and each page, and decide, "Oh, less than ten seconds, we're not going to 
count that as a valid look. We just think that was navigating from one page to the next. " You 
have to do all that before you have data that are even meaningful, even to answer the question, 
"What fraction of the pages does the student look at in the book?" 

We're not using standard procedures to do this. We have to write our own programs. We 
have to write, not just a little program: we have to figure out how to write it in a generalizable 
way so that we can use it to analyze the videos as well as the textbook. A significant part of the 
challenge is to figure out how to avoid double-counting and how not to lose 1,500 students' 
data. It's not all organized for you. (Pritchard, MIT). 

Ken Koedinger, Professor of Human-Computer Interaction and Psychology, Carnegie Mellon University, 
and Director of the Pittsburgh Science of Learning Center (PSLC), drew parallels between the challenges 
of processing data from traditional instructional data systems and online courses, and MOOC platform 
data. He asserts that it is easy to underestimate the value of "keystroke" or "clickstream" data from a 
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well-designed instructional technology system in understanding the nature of cognitive processes, and 
easy to overestimate the value of a poorly designed system. He remarked that knowing when a 
participant has started and stopped a video is of little value unless this information can be combined 
with information on what students have learned. Currently, the latter is most commonly collected 
through "structured entry in a graphical interface," that is, responses to multiple choice or menu-based 
questions, or entries into a table. 

Koedinger identified additional current limitations to the use of data from online platforms. For 
example, when recording activity of users of simulation software, most systems only log the state of the 
simulation at any one time instead of asking what the learner does to change the simulation. He also 
observed that there has been significant progress in development of software tools to help code the 
content of discussion forums, but that privacy issues are more of a challenge for analysis of discussion 
forum data compared with clickstream data. 

Whitmer pointed out the need for standardization of the current data formats. He suggested that either 
the MOOC platform providers take on the task of standardization, or that a combination of researchers 
and programmers develop a format for everyone to follow. Whitmer also observed that programmers at 
multiple sites are building data logging routines from scratch, and platform providers are each writing 
their own data queries, often duplicating effort. A repository for such queries could allow sharing and 
improved efficiency. Whitmer pointed to the work of the IMS Global Learning Consortium in developing 
Caliper, a common learning analytics framework that can be used to capture and present learning 
activity metrics from a variety of systems (see IMS Global Learning Consortium, 2013) . Koedinger 
hypothesized that the current situation arises because of a "not invented here" culture among 
programmers that steers them towards building their own systems rather than applying or adapting 
what has already been created. 

Baker suggested that MOOCs need a "Science of Learning Center" that stores both restricted access data 
sets from MOOC platforms as well as open access reference datasets for any researcher to use, guides 
for data logging formats, and analytical tools. The existing Pittsburgh Science of Learning Center (PSLC) 
already addresses similar data issues for online learning systems through its DataShop, a "data analysis 
service for the learning science community." (Baker was formerly the Technical Director of DataShop 
and is currently a Scientific Advisor). According to Koedinger, PSLC was established with National Science 
Foundation funding ten years ago to collect data from instructional data systems and online courses, 
and to use these data to answer research questions. DataShop acts as "a central repository to secure 
and store research data" and provides "a set of analysis and reporting tools" 

( https://pslcdatashop.web.cmu.edu/about/) , and recommends a standard format for logging data to 
allow ease of analysis across multiple online learning systems. For example, the data logging and 
formatting recommendations specify what information is useful to collect about user activity on the 
platform and how to report data in commonly recognizable spreadsheets. Available tools facilitate data 
visualization and statistical modeling. Koedinger is currently in discussions with faculty members from 
MIT, Stanford, and Teachers College regarding the creation of a cross-university collaboration for MOOC 
data, starting with a discussion of what are the current problems and eventual goals for use of these 
data. 

The Anyscale Learning for All ( ALFA group ), led by Una May O'Reilly and part of MIT's Computer Science 
and Artificial Intelligence Laboratory (CSAIL), appears to be putting a stake in the ground with regards to 
addressing some of the MOOC data issues with the development of MOOCdb (see Veeramachaneni et 
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al., 2013 ). Dernoncourt et al. (2013), of the ALFA group demonstrate how MOOCdb allows for analysis of 
course data from both the Coursera and edX platforms. However, it remains to be seen whether other 
groups will agree to play in this sandbox or will continue to invent their own tools. And until the tools 
developed by such groups can do more than provide attractive pictures of descriptive data, the frontiers 
of knowledge about effective pedagogy will not be breached. 

To realize the full potential of research that can be conducted with the big data from MOOC platforms, 
Saxberg, Chief Learning Officer at Kaplan Inc., and David Niemi, VP, Measurement and Research, also at 
Kaplan Inc., suggest that multiple players must engage with each other and strategically plan what data 
are needed and how the data will be managed. 

It's kind of a nontrivial thing, actually, to spell out all the data you want in the beginning, and 
have the system built around that because it's hundreds of potential variables. You're going to 
have to think about who is every conceivable person that wants to do something with the data, 
get all of those guestions. (Niemi, Kaplan Inc.). 

You need someone with a deep measurement capability, somebody with deep research 
capabilities who knows what kind of analysis they're going to want to do at the end, somebody 
with strong learning science instructional design capabilities because they're going to know what 
sorts of things they're going to want to try. Then you need your IT people at the table... (Saxberg, 
Kaplan Inc.). 

Beyond the planning for production of workable platform data, analysts must be trained in various 
aspects of handling data, a multi-step process according to Baker. First, the analyst must learn how to 
store data in accessible formats. Second, the raw data needs to be transformed into meaningful 
variables through "feature engineering." Baker emphasized that this process is hard to teach and that 
there are few courses available on these techniques. A third step involves identifying patterns in the 
data. Analysts must also master broader theoretical questions about how data mining can be used to 
understand learning processes, and consider issues such as privacy, societal values, and potential impact 
of the research. 

Issues of data privacy, access, and sharing are as yet unresolved in the relatively new legal territory of 
MOOCs (see Dillenbourg et al., Eds., 2014). For example, to whom do the platform data belong and who 
can have access without violating FERPA requirements? We found some interviewees more enthusiastic 
than others regarding the prospect of sharing MOOC data among institutions for research or other 
purposes, with or without a license fee, but many pointed to significant confusion regarding whether 
any or all MOOC participants are covered by FERPA or other regulations given that the courses are free 
and non-credit-bearing, that most students are not enrolled at the MOOC provider's campus, and that 
many participants are international. But as Means from SRI pointedly noted, "There's a lot of 
misinterpretation of FERPA and a lot of times it's actually a really convenient excuse for not providing 
information to people." 

It is apparent that while the potential for MOOCs to contribute significantly to the development of 
personalized and adaptive learning is high, the reality is far from being achieved. A great deal of 
coordination and collaboration among researchers, instructional designers, and programmers will be 
necessary to result in meaningful improvements to teaching and learning. 
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Resource Requirements and Costs of MOOCs 


In this section we first review the resources required to produce and deliver, or "consume" MOOCs. 
Subsequently, we summarize the actual costs of MOOCs from the perspective of the producer (i.e., the 
college, university, or museum, as opposed to the platform provider or participant) including the results 
of cost analyses we conducted of MOOC production and delivery at four different institutions. The 
details of each analysis are available in Case 10. Case 11, Case 12 and Case 13 . For MOOCs that are 
delivered via third party platforms, there are often significant, additional costs to the platform provider 
which may or may not be passed on to the MOOC producers, depending on whether there is a direct 
charge for the platform services or a revenue-sharing agreement. 

The major cost drivers in MOOC production and delivery are: the number of faculty members, 
administrators, and instructional support personnel participating in the process; the quality of 
videography; the nature of the delivery platform; technical support for participants; programming for 
special features such as computer code auto-graders, virtual labs, simulations, or gamification; and 
analysis of platform data. MOOC production teams that were described to us seldom included fewer 
than five professionals and, in at least one instance, over 30 people were involved. Course length was 
not a reliable predictor of costs. 

MOOC production resource requirements 

Personnel 

At most institutions, MOOC development has been a team effort, sometimes including administrators, 
instructional designers, instructional technologists, programmers, project managers, videographers, and 
evaluation specialists in addition to the faculty members and teaching assistants (TAs). In contrast to the 
often solo endeavor of developing a traditional course, MOOCs entail what Schneider at Stanford 
University described as "a distributed sense of responsibility for the design of the course." A few 
institutions have created offices of online learning to facilitate the development of MOOCs and other 
online and blended courses. A committee may be established to receive and review project requests 
from faculty members and to award seed grants. For example, at Stanford University an entire unit 
entitled the Office of the Vice President for Online Learning ( VPOL ) has been created within the last 18 
months. By October 2013, VPOL employed 30 people (four administrators, three researchers plus 
several doctoral students and post-doctoral fellows, seven in platforms and engineering, five in course 
design and development, five in course and media production, and one in communications). According 
to Collier, this team is working on over 100 projects, a quarter of which are MOOCs. Seed grants of 
$25,000 are provided to faculty members for innovative projects in online and blended learning. Such an 
office alone would require several million dollars per year to operate, before considering the efforts of 
faculty members and TAs. Other institutions relied on expanded Centers for Teaching Excellence or 
Educational Innovation to help design and develop MOOCs. 

All interviewees who had been involved in the development of a MOOC reported the effort being two to 
three times greater than creating a traditional course. Instructors typically spent several hundred hours 
over several months preparing and re-purposing course materials, and practicing lecture delivery prior 
to video-taping; several days on actual shoots; and one to two days reviewing the finished video. To 
create one hour's worth of MOOC video-lecture required three to ten hours of preparation according to 
several faculty members, the lower end of the range being in instances where the materials were being 
re-purposed from existing lectures. To create ten minutes of voice-over-PowerPoint video required six 
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to eight hours according to an interviewee at a private university. See, for example, Cima's (2013) 
account (pp. 15-17) of the time and effort required to prepare and deliver the MOOC Introduction to 
Solid-state Chemistry. Cheal at San Jose State University, among other interviewees, likened the effort 
involved in creating a MOOC with writing a textbook in a team. 

At some institutions faculty members are granted a course release and/or paid stipends ranging from 
$3,000 - $15,000 per course while developing and delivering a MOOC, but the opportunity costs of the 
instructor's time are likely to be higher in many instances. We frequently heard estimates in the order of 
400 hours of faculty member time per MOOC developed. While we did not hear of instructors being paid 
per additional student, Silverman at NYU points out that the union contract governing faculty member 
compensation at his institution requires that instructors be paid $50 - $250 per student above a fixed 
student load. Clearly such clauses would need to be revisited in situations where an instructor offered a 
MOOC that attracted large enrollments. 

Courses that have been developed and significantly refined over several years may cost substantially 
more than those created over a period of a few months. Pritchard, from MIT, estimates that three to 
four post-doctoral-fellow years have been invested in developing and refining his Mechanics Review 
course which has been offered as a MOOC three times. The post-doctoral fellow alone would account 
for $200,000-$250,000 at a national average salary and benefits rate. In addition, many hours of 
undergraduate and faculty time were committed to the course development. 

MOOC platform costs 

It appears that platform costs for MOOCs offered by Coursera and Udacity are often subsumed into 
revenue-sharing agreements (see, for example, the Coursera contract with University of Michigan; 
Young, 2012) , in which the platform provider takes responsibility for hosting the courses and drawing in 
participants, but also keeps most of the revenues earned from those participants who purchase 
certificates or other services. Coursera's arrangements with state university systems vary but, in one 
example, the State of Tennessee was asked to pay $3,000 per course for use of the Coursera platform, 
plus $25 per student (see Rivard, 2013). One community college paid Udacity a fixed sum of $30,000 for 
its role in helping to develop and deliver a MOOC on the Udacity platform and providing data on 
participant activity and performance. edX, a nonprofit organization, offers a "spectrum of full service to 
self-service" options with respect to MOOC development and delivery with varying arrangements for 
revenue sharing (see Kolowich, 2013e ). Partners may commit to negotiable cash or in-kind contributions 
(e.g., personnel time, code development, collaboration on research) in order to be supported in their 
use of the edX platform and for their courses to be hosted on the edX servers. Optional services for 
assistance in developing and delivering the courses are for additional fee. These services might include 
instructional design training or video production. A typical contribution might be $250,000 per course 
and $50,000 for a re-run, plus a revenue-sharing arrangement. One interviewee estimated the costs to a 
public university for the use of a platform developed by a small, for-profit entity at $20,000 per course. 
These fees covered the programming costs of creating the platform, "onboarding" courses, hosting, 
monitoring, managing and maintaining them, data provision, and technical support for course 
participants. Platforms such as Open edX or Canvas Network are available free but the MOOC provider 
must have the capacity to create and upload compatible materials, manage the course delivery and 
attract participants. 
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Videography 


Many MOOCs include an hour or more of video per week and we found that videography costs were 
highly variable. We conducted a detailed estimate of professional videography costs which resulted in a 
sum of $4,300 per hour of finished video (see Appendix IV ). Some institutions use in-house video 
production teams which cost significantly less (see Case 13 ), while others pay substantially more than 
our estimate (see Case 12) . We even heard of two cases where a professor established his own 
recording studio in a home basement. 

Assessment 

Assessing students at scale also requires a significant investment of resources, first to develop 
assignments that can be graded in large numbers and, secondly to devise peer-grading or auto-grading 
mechanisms or to provide enough TAs to cope with the quantity of assignments to be reviewed. In most 
cases, multiple choice or short answer tests are devised for MOOCs. All questions and answers must be 
thoroughly "debugged" before use. In some academic departments, a large bank of questions is 
collected so that students can be assigned similar questions at random to allow multiple attempts at 
mastering a concept or problem-solving strategy, and to help avoid cheating. At the University of 
California, Berkeley, four auto-graders have been developed, requiring an estimated 1,000 hours of 
engineering time, to allow MOOC participants' computer code assignments to be automatically 
assessed. A few institutions were in the process of hiring a Director of Assessment or Director of 
Learning Outcomes, or had recently done so. 

Obtaining copyright permissions 

Additional costs may be incurred for use of copyrighted materials, both for the personnel time involved 
in obtaining permissions and for fees to cover use of the material. Some institutions worked to 
substitute copyrighted images and text with open or self-generated artifacts. 

Refreshing MOOCs to be re-run 

After the first iteration of a MOOC, almost all institutions have made adjustments to the design and 
content before re-running. These vary in scope and effort, as do expectations for the need to refresh 
content on a regular basis. White expected that the computer science MOOCs offered by Georgia Tech 
would be updated every three years on average, while Li Fengliang of Tsinghua University insisted that 
MOOC content should be changed every year. 

MOOC delivery resource requirements 

While some MOOCs are designed to run without faculty or other ongoing intervention, many online 
courses involve significant instructor and TA or Graduate Assistant involvement. Several faculty 
members reported many hours unexpectedly spent on intensive "debugging" and mid-course 
corrections during the first offering of a MOOC. More routinely, instructors estimated spending at least 
20-30 minutes per day participating in the discussion forums. MOOCs often have one or more TAs 
assigned to monitor these forums and to respond to participant questions. Fox at the University of 
California, Berkeley estimated 40-50 hours per week of TA time needed to support an engineering 
MOOC. Additionally, some participants become de facto "community TAs" by being especially helpful in 
responding to the questions of others participants. 
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Varying amounts of effort are being made to evaluate the success of MOOCs, with a few institutions 
assigning internal evaluators to assess their impact or requesting that faculty members conduct their 
own reviews and submit a formal report to other faculty members on lessons learned. Analyzing the 
MOOC platform data beyond what is readily available in the participant activity dashboards in order to 
improve future iterations of MOOCs can be time-consuming and expensive. 

Other university or college services required to support MOOCs 

In addition to the direct costs of producing and delivering MOOCs, Cheal (2012) provides insights into a 
plethora of additional considerations for institutions engaging with MOOCs. For example, MOOCs can 
only attract massive audiences if they are sufficiently marketed. While the platform providers such as 
Coursera, edX, and Academic Partnerships fulfill these marketing and communications functions for 
their partner institutions, those institutions using more “do-it-yourself" platforms must find suitable 
advertising channels. 

Computing and Internet services for on-campus students participating in MOOCs may need to be 
increased or upgraded, for example, retrofitting buildings to provide enough bandwidth capacity for 
many students to simultaneously download video. Institutional websites and learning management 
systems are likely to need adjusting to provide an access point to MOOCs. While most major platform 
providers offer technical support to participants, students who are enrolled at a college or university 
and are participating in online courses, including MOOCs, often contact their own institution's help desk. 
This may require the platform provider to work with help desk staff to prepare them for inquiries, and 
may require additional staff. 

A variety of administrative offices are likely to be involved in activities such as establishing contracts 
between the institution and online platform provider, and between the institution and its faculty 
members to address intellectual property rights, revenue sharing, faculty compensation, and workload 
issues. Compliance with disability regulations in MOOCs must be regularly audited and enforced, and 
accommodations made, for example, extra time on quizzes and exams for students with learning 
disabilities. 

For institutions providing credit for MOOCs, the student admissions, registration, billing, authentication, 
and crediting systems need to be aligned with platform enrollment procedures (see Cheal, 2012). If 
prerequisites are required for credit-earning participation in a course, a system must be developed to 
handle large numbers of students. 


MOOC "consumption” resource requirements 

While the above considerations pertain to institutions that are producing MOOCs, there are also many 
resource requirements for institutions that are integrating MOOCs created by others into their own 
courses. Bacow, President Emeritus of Tufts University, identified the following considerations: 

• Search cost: time required to find a MOOC that is relevant to the instructors' existing course. 
Comparable to putting a course together from books, articles, and other materials. 

• Familiarity with material/adaptation cost: time required to become familiar with the material as 
it is being delivered so that it can be interpreted and translated for instructor's own objectives 
and student population. 

• Licensing fees to use the material. 
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• Finding material to supplement the MOOC in order for it to fit the existing curriculum and 
student needs. For example, if you are using a statistics MOOC in an environmental statistics 
course, you would want different materials than for a course on statistics for engineering. 

• Assessment: creating your own exams and problem sets. 

• Space cost: to bring students together, assuming the MOOC is being used in a blended learning 
course. If you are a traditional residential institution, it is difficult to explain to a student why you 
should be paying $50,000 for tuition to just sit in your dorm room and look at it online. So you 
have to offer some personal contact. 

• Professor's salary for the course. 

• Certification costs: giving credit, recording it, being able to indicate that students completed the 
course. 

• Cost of technology: if you are reliant on technology to deliver a course, you have to have 
redundancy. You cannot afford to let anything go wrong. You have to have people to resolve 
technology problems immediately. (Bacow, President Emeritus, Tufts University). 

Costs of MOOC production and delivery 

Using the ingredients method (Levin & McEwan, 2001) we estimated the actual costs of MOOC 
production and delivery at four different institutions (see Case 10, Case 11, Case 12, and Case 13) . 
Personnel costs alone ranged between $29,000 and $244,000 per MOOC, depending on the number of 
people involved in the process, the quality of video production and the amount of time dedicated. 
Although it would be more accurate to estimate costs of facilities, equipment, and overhead 
individually, we rely on Levin and McEwan's (2001) observation that, for educational interventions, 
personnel costs typically account for 75% of total costs. We estimate total costs per MOOC of $39,000 - 
$325,300, as shown in Figure 3. Table 5 additionally indicates the length of the MOOCs involved and, in 
two cases where completion data were available, cost per completer. 


Figure 3: Estimated Costs of MOOC Production and Delivery at Four Institutions (low end estimates) 


Large Midwestern University 
American Museum of Natural History 
University of Manitoba 
Teachers College, Columbia University 



By comparison, Bates at Contact North currently estimates the costs to develop a regular three-credit 
online course delivered on a learning management system at $35,000 -$50,000 (A. Bates, personal 
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communication, April 29, 2014; Bates & Sangra, 2011, pp. 163-166). Laster, of McGraw-Hill Education, 
believes that the initial investment of time and resources to develop MOOCs is likely to be three to four 
times greater than for developing a face-to-face course based on the same content and learning goals. 

Many interviewees provided their own rough estimates of MOOC production and delivery costs, but 
acknowledged that these did not rely on rigorous time-use studies or actual cost analyses. The rough 
estimates reported by interviewees ranged from as low as $5,000 per course at a university where the 
online course development infrastructure was already in place and existing materials could be re- 
purposed for MOOCs, to as high as $1.2 million at a university that is new to online course delivery and 
is building sophisticated auto-graders, virtual laboratories and simulations, which may eventually be 
used in multiple courses (see Box 10) . The community colleges in our sample estimated spending in the 
order of $75,000 per MOOC because they were working from grants of $50,000 and contributing in-kind 
efforts. The original projected costs for Georgia Tech's highly publicized Online M.S. in Computer Science 
(OMSCS) program are publicly available . We provide a review of these costs and the staff/student ratios 
for the OMSCS courses in Case 5, along with our concerns over the feasibility of these plans. 


Table 5: Estimated Costs of MOOC Production and Delivery at Four Institutions 


Institution 

Type of MOOC 

Length of MOOC 
(weeks) 

Total estimated costs 
per MOOC 

Costs per 
completer 

Teachers College, 
Columbia University 

xMOOC 

8 

$38,980 

$74 

University of Manitoba 

cMOOC 

12 

$65,800-71,800 

* 

American Museum of 
Natural History 

xMOOC 

4 

$104,620 

$272 

Large Midwestern 
University 

xMOOC 

5-8 

$203,770 -$325,330 

* 


* Completion data were not available for these MOOCs. See Cases 10, 11, 12, and 13 for sources. 


The question for MOOCs over the longer term is whether variable costs can be contained by automating 
functions and substituting instructional support provided by expensive faculty members with less costly 
TAs, part-time instructors, or peer-to-peer learning and assessment. In 2012, just as MOOC-mania was 
beginning to build, Bowen et al. provided an illustration of how instructor costs could be reduced in 
online and hybrid courses by using part-time instructors and altering course configurations. Some of 
these ideas could be adopted for MOOC delivery. 

Many MOOC developers anticipate that the costs of re-running a MOOC will be substantially lower than 
the costs of initial development. In Case 10 we show that the costs of re-running the first cMOOC, 
Connectivism and Connected Knowledge, were 38% lower than the costs of the initial offering. However, 
given the intense level of instructor involvement in cMOOCs, this is unlikely to be a useful predictor for 
xMOOC re-runs where instructor involvement may be minimal or absent. James at Mt. San Jacinto 
Community College expected that the re-run costs for the college's xMOOC, Crafting an Effective Writer, 
would be minimal, perhaps less than $1,000, compared with her estimate of $75,000 for the initial 
development and offering. Such assumptions should be rigorously tested through careful cost analyses 
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and we recommend that, going forward, MOOC producers attempt to document these re-run costs to 
help assess the sustainability of MOOC production. 


Box 10: 

One Interviewee's Estimates of MOOC Production and Delivery Costs 

[If developing a course] for a traditional class for 30 (or 300) students costs $100,000 then, on average, 
the first time you [turn a class into a MOOC] it might be $250,000 to [produce it] and another $75,000 - 
$150,000 to deliver the first time. 

Subsequent delivery would be about half that - call it $50,000 - $75,000. Refreshing could be anywhere 
from $25,000 to $75,000 depending on the nature of the class and nature of the changes. 

However, ...there are other complexities. The first Biology course in a track might require a protein 
modeler which costs $75,000-$100,000 to build. Once built, we can use it for other Biology courses. So 
those subsequent courses in the same sequence should be substantially less expensive to create and 
deliver. Some courses could cost as much as $750,000+ to $1.2 million to produce - these "monster" 
classes would spawn a number of shorter courses. 


While our cost analyses help shed light on the costs of MOOC production and delivery, several important 
questions remain unanswered, in addition to whether repeat offerings will cost less than the first 
delivery. While the cost numbers seem high and many instructors perceived greater expenditure of 
effort to create a MOOC as compared to developing a face-to-face or regular online course, we have no 
documentation to prove this. If we wish to answer the question of whether MOOCs are a cost-effective 
means to deliver education, we must be able to compare the costs of MOOCs to the costs of the 
alternative delivery mechanisms, as well as the effectiveness of each alternative with respect to a 
common outcome of interest, such as increasing participants' level of knowledge or skill in a specific 
subject area. 
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Case 10: 

Cost Analysis for Development and Delivery of 
Connectivism and Connected Knowledge (a cMOOC) 

Connectivism and Connected Knowledge ( CCK08 ), the first course to be dubbed a "MOOC," was 
developed and delivered in 2008 by George Siemens and Stephen Downes. It was offered at the 
University of Manitoba to enrolled students for fee and for credit and also as a free, non-credit-bearing 
course to any other interested participants. Twenty-five students formally enrolled in the 12-week 
course and another 2,300 participated. The course has been re-run three times since. 

Siemens estimated the time burden for CCK08 development and delivery as follows: 

• Course design and development: 100-150 hours over two months; 

• Course delivery: 70 hours per week for the first 2-3 weeks (interacting with students and posting 
on discussion forums or writing blog posts to summarize discussion and activities), tapering 
down to 30 hours per week in the twelfth week. 

At the lower end of Siemens' estimates, the total number of hours would be around 715. At the high 
end, the time commitment could have been as much as 770 hours. We estimate costs at each end of the 
range. 

Downes estimated his total time commitment for CCK08 at 88-108 hours: 

• 20-40 hours in programming time to make adjustments to the gRSShopper course aggregation 
software that he had developed over many years; 

• 20 hours setting up the course website; 

• 4 hours per week during course delivery to maintain the site and prepare audio archives. 

Using U.S. national average salary and benefits rates for public postsecondary faculty members and 
public sector research scientists, the costs of personnel time to replicate CCK08 ranges from $49,400 to 
$53,800. No technology support, learning designers, or teaching assistants (TAs) were utilized in the 
development and delivery of CCK08. We do not estimate costs individually for facilities, other 
equipment and overhead but assuming personnel costs amount to 75% of total costs (see Levin & 
McEwan, 2001), the total cost estimate to replicate the cMOOC CCK08 would be $65,800 to $71,800 as 
shown in Table 6 below. 

Re-runs of CCK08 required less design and development time. Additionally, with better course 
management software, weekly delivery time for CCK12 fell to 30 to 40 hours for the first two to three 
weeks. Some repeat students self-selected as TAs and reduced the instructors' time burden by helping 
manage the forums, responding to inquiries and providing guidance to new students. (Downes 
estimates that around 100 participants from CCK08 returned for CCK09 ). Set-up time for the course 
website dropped from 20 hours to 4 hours. Twitter and Diigo were incorporated as new tools. 
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For Siemens, we estimate the total time commitment for a CCK08 re-run at 284 hours: 

• 20 hours to "refresh" the course design and resources before a new launch; 

• 28 hours per week in delivery for the first 3 weeks; 

• 20 hours per week in delivery for the remaining 9 weeks. 

For Downes, we estimate the total time commitment for a CCK08 re-run at 72 hours: 

• 4 hours website set-up; 

• 20 hours adjusting gRSShopper to accommodate new tools; 

• 4 hours per week maintaining the course site. 


Table 6: Estimated Replication Costs for the First Run of CCK08 and Re-run 


Ingredient 

First run 
low estimate 

First run 
high estimate 

Re-run 

estimate 

Instructor 1 

$44,756 

$48,199 

$17,777 

Instructor 2 

$4,597 

$5,642 

$3,761 

TAs 

- 

- 

$9,015 

Facilities, equipment, overhead 

$16,451 

$17,947 

$10,184 

Total 

$65,804 

$71,788 

S40,737 


The possible range of time committed by the self-selected TAs could be very wide and we do not have 
reliable data to use. However, given the substantive role played by these participants, the opportunity 
cost of their time should be considered when replicating a MOOC. We use an estimate of 350 hours 
total, using an assumption that the TAs collectively replace the reduced hours in Siemens's delivery 
time. We use the lower end of Siemens' delivery time estimates to be conservative. Total estimated 
costs for the re-run are $40,740, 38% lower than the low estimate for the first run. 

Siemens indicates that planning time for a cMOOC is greater than for a face-to-face course because 
there is less room for correcting errors "on the fly.” Delivery time is far greater: with a face-to-face 
course, there are fewer learners resulting in less writing to be read, and less time required to synthesize 
contributions. With over 2,000 students in a MOOC, Siemens observes that the instructor "can be active 
in a huge range of conversations." 


Sources: George Siemens, formerly at Athabasca University, and Stephen Downes, National Research Council. 
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Box 11: 

What is gRSShopper Anyway? 

“It is a Swiss-Army-Knife-of-an-application that is optimized to support distributed network resources. 
It's a blogging tool, it's an RSS reader, it's an e-mail mailing list. Websites such as blogging sites or 
discussion sites can create a data representation of their content called "feeds." A feed is a machine- 
readable version of the content of the site. When the site is updated, the feed is updated. An RSS reader 
accesses a specific type of feed called an RSS feed. It retrieves that feed and displays the content in a 
centralized location for the user. People use an RSS reader to read a number of different websites all in 
a single location. gRSShopper retrieves feeds from students participating in our MOOCs. gRSShopper is 
more than just an RSS reader because it can publish the aggregated feeds as a web page or as an e-mail 
newsletter. Our MOOC participants write in their own blogs which might be on Blogger, WordPress, 
Tumblr, Posterous, and so on. gRSShopper aggregates these feeds, organizes them, puts them in a 
searchable database and automatically sends out a newsletter of all the posts that participants made in 
the previous 24 hours. 

To participate in one of our courses, people just read an e-mail, click on any link shown and go directly 
to some other person's website to look at what they posted on their website, and perhaps comment on 
that website. They don't even have to come to our course website, they can simply interact with other 
participants in the course. Most online courses are site-based, centralized, and participants need to visit 
the site to access resources. gRSShopper allows you to stay where you are. It will come get you. 

Source: Stephen Downes, National Research Council. 
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Case 11: 

Costs of MOOC Production at a Large Midwestern University 

Before entering the MOOCspace, this university already had an established infrastructure and personnel 
for the development of online courses. Once the decision was made to offer MOOCs, a small number of 
faculty members with prior "media experience" were invited to develop and deliver five- to eight-week 
MOOCs, primarily to showcase the university and engage new audiences. Each faculty member was 
assigned a design and support team of five to six people to help in the design and production of the 
MOOC. Each team included a project manager, instructional designers, instructional technologists, and a 
liaison to the video production team. Additional personnel supervised the design and support teams, 
and provided programming capacity, overall project management, evaluation, and administrative 
services, including campus events to increase knowledge about MOOCs. As a routine part of the project 
management function at this university, detailed time logs are kept by each design team member so 
that costs for these personnel can be tracked accurately. We used the cost estimates provided by the 
university for these personnel in our analysis because we did not obtain enough detail regarding the 
personnel ingredients (e.g., specific role, level of experience, highest degree of education) to allow us to 
assign prices ourselves. Faculty member and teaching assistant (TA) time were not logged but estimates 
were obtained either during or after MOOC production. We estimated costs for faculty members and 
TAs ourselves, using national average salary and benefits rates for postsecondary public institutions. For 
the first three MOOCs created and delivered, the time burdens and resulting cost estimates are shown 
in Tables 7 and 8 respectively. 

Table 7: Range of Hours Spent per MOOC on Design, Production, and Delivery at a Large Midwestern 
University 



Low 

High 



Design team 

200 

500 



Video production team 

700 

900 



Platform/technical issues 

150 

155 



Faculty member 

90 

220 



TA 

0 

650 



Total 

1,140 

2,245 


Table 8: Range of Estimated Personnel Costs per MOOC for Design, Production, and Delivery at a Large 
Midwestern University 



Low 

High 



Design and support team 

$70,000 

$125,000 



Computer programming unit 

$0 

$15,000 



Management (avg. across 3 MOOCs) 

$77,200 

$77,200 



Faculty member 

$5,630 

$13,770 



TA 

$0 

$13,029 



Total personnel costs 

$ 152,830 

$244,000 
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The faculty time burden was relatively low because the dedicated design and support team took on 
much of the task of course design and development. Design team time varied depending on the 
complexity of the learning activities. For example, in one MOOC, participants were required to upload 
artifacts and report data which were automatically analyzed. Overall results were shared out to all the 
MOOC participants. The amount of time required for platform and technical issues such as uploading 
content and “getting things to work on Coursera" dropped as the team built experience with the 
platform. Note that the costs of facilities, equipment and overhead are not estimated here but, if we 
assume that personnel costs account for 75% of total costs (Levin & McEwan, 2001), the range of total 
costs per MOOC would be $203,770 - $325,330. We note that salary levels at this geographical location 
may be lower than national averages so that costs for the non-teaching personnel could be higher on a 
national average basis (in the order of a few thousand dollars). 


Source: Evaluator at a large Midwestern university. 
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Case 12: 

American Museum of Natural History 
MOOC Initiative: Resource Requirements and Expected Benefits 

Between September and December 2013, the American Museum of Natural History (AMNH), as an 
invited partner in the Coursera Teacher Professional Development program, serially delivered three 
four-week long MOOCs targeted at science educators. Planning efforts began in Spring 2013 and 
involved a team of museum professionals who had significant previous experience in developing and 
delivering online education, including Seminars on Science, an online education program for science 
teachers. The core MOOC production team comprised a project director, a project manager, an in-house 
video producer, an educational technologist, and a senior administrator who also served as one of the 
MOOC instructors. 

While the museum had already previously developed many digital resources including science-content 
videos and educational essays on science topics, MOOCs presented a new challenge to develop lecture- 
based videos with "talking heads" or voice-over PowerPoint presentations, multiple-choice quizzes, 
peer-graded assessments, and pre- and post-course surveys. In addition, large amounts of MOOC 
participant data became available for analysis and interpretation. The personnel effort associated with 
the production and delivery of the three MOOCs, as well as processing the course-related data are 
summarized in Table 9, based on estimates of time use provided by the AMNH team. 

The project manager and project director spent the equivalent of 25 and 11 entire workweeks 
respectively on the project, while the instructors spent, on average, about six workweeks each, shooting 
videos and developing, adapting, or reviewing course content. The core team met once or twice a week 
for one to two hours to plan, design, execute, and review the MOOC production and delivery. The 
teaching assistant (TA) spent a total of 400 hours managing the discussion forums, processing survey 
responses, and reviewing the platform data. 

Using national average salaries and benefits rates for personnel, wherever possible at similar positions 
in postsecondary institutions to allow comparability with the other MOOC costs we present, we 
estimate the personnel costs to develop the three MOOCs created by AMNH at $235,000, or $78,470 
per MOOC. If we assume that personnel costs account for 75% of total costs once facilities, equipment, 
and overhead are included (Levin and McEwan, 2001, p.53), we estimate total costs at $104,620 per 
MOOC. Of the total 39,685 participants who initially enrolled in the three MOOCs, 1,155 earned a 
certificate of completion (scoring 72%-84% of possible points) or a certificate of distinction (scoring 85% 
of possible points). Costs per completer for the MOOCs amount to $272. 

Because we are mostly using postsecondary personnel prices, we assume a 1,560-hour year, a rate of 
28.5% for benefits, and that all employees are in-house. We expect that actual costs for AMNH would be 
significantly higher given that New York City costs are above the national average, benefits rates at the 
museum are 40.5% for full-time staff, and several of the personnel were hired as consultants at hourly 
rates. For example, the HTML writer cost almost three times more than the national average estimate, 
and costs of video shooting and editing amounted to 5.5 times more. The costs of the platform, 
captioning, content hosting, and analysis of user data to populate the data dashboard were assumed by 
Coursera. 
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Table 9: Personnel Hours Spent by AMNH to Develop Three MOOCs 


Ingredient 

Hours 

Senior management 

125 

Project director 

454 

Project manager 

980 

Instructors 

910 

Educational technologist 

174 

TA 

400 

Evaluation expert 

16 

Graphic designer 

350 

Video producer 

293 

Video shooter 

63 

Video editor 

210 

HTML writer 

30 

CSS Writer 

10 

Legal personnel 

13 

Marketing personnel 

12 

Business manager 

5 

Total for 3 MOOCs 

4, 045 

Per MOOC 

1,348 


AMNH will be offering the three MOOCs again throughout 2014 and while adjustments are being made 
to each course, initial costs of production can be amortized over the multiple repeats, allowing a "run 
rate" to be established for the ongoing delivery and maintenance of the courses. 

In addition to assessing success of the MOOC initiative in terms of course completions, other indicators 
of impact are being monitored. For example, survey responses and follow-up interviews with consenting 
participants will indicate whether teachers report learning new content and using museum materials 
and teaching strategies in their own classrooms. The number of people who visit the museum online or 
on-site, the number of new international and domestic users of AMNH content, (e.g., the science media 
videos or Science Bulletins that are recommended in the MOOCs), and the number of people who enroll 
in Seminars on Science are being tracked and compared to pre-MOOC activity. 

Sources: 

Dr. Ro Kinzler, Senior Director, Science Education, American Museum of Natural History, NY; 

Dr. Robert Steiner, Director, Online Teacher Education Programs, American Museum of Natural History, NY; 

Maria Janelli, Senior Manager, Online Teacher Education Programs, American Museum of Natural History, NY. 
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Case 13: 

Time-by-Task and Cost Analysis for Big Data in Education Development and 

Delivery 

Big Data in Education was an eight-week MOOC delivered on the Coursera platform between late 
October 2013 and mid-December 2013. Ryan Baker, a faculty member at Teachers College, Columbia 
University, developed the course, adapting it from a 16-week on-campus version usually taught to 
classes ranging in size from eight to fifteen students. While Baker's initial target enrollment for this 
specialized course on educational data mining and learning analytics was 200 participants, 48,058 
individuals registered for the course, 2,650 watched the final lecture, and 526 completed the last 
assignment. Planning and preparation for the course began in mid-March 2013. Big Data in Education 
was free, open to any participant, and non-credit-bearing. Participants who completed the course 
earned a Statement of Accomplishment. 

Baker kept track of time and tasks related to the MOOC in an Excel spreadsheet from June (when our 
study began) to the end of December 2013. Hours spent on activities prior to that date were estimated 
Total time logged plus time estimated was 176 hours, with the heaviest burden falling during the first 
three months of planning and preparation of materials, the month prior to launch, and the first few 
weeks of course delivery (see Figure 4). Time spent on various tasks is shown in Figure 5 and included: 


• Creating course materials such as slides, assignments, and quizzes (58 hours); 

• Set-up and video-recording using ScreenFlow software (46 hours yielding 6 1/2 hours of finished 
video used in the MOOC); 

• Planning, bureaucracy, and coordination with Coursera, the teaching assistant (TA), and the 
course production team (37 hours); 

• Participating in the forums and responding to participant e-mails (26 hours); 

• "Debugging" slides, assignments, and quiz questions during the course (7 hours); 

• Open office hours (3 hours). 


Figure 4: Faculty Member's Hours by Month for Development and Delivery of Big Data in Education 
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Over the 41 weeks of preparation and delivery, Baker spent an average of 4 1/2 hours per week on the 
course with a high of 22 1/2 hours in one week and a low of zero hours in another. In addition to Baker, 
several other personnel worked on the MOOC: 

• A teaching assistant spent approximately 15 hours per week over 16 weeks for a total of 240 
hours, with the heaviest burden falling during the three weeks prior to course launch and during 
the first three weeks of course delivery. Tasks included: coordinating among faculty member, 
video team, and Coursera's course coordinator; checking that the uploaded videos were 
working; posting assignments and "inline" quiz questions (which are embedded in the videos); 
and participating in the discussion forum. 

• Seven individuals from the Educational Data Mining Laboratory at Teachers College read and 
participated in the discussion forums. Two were M.A. students, three were Ph.D. students and 
two were post-doctoral fellows. We estimate 2 hours per person per week over the 8 week 
period for a total of 112 hours. 

• A senior administrator coordinated the production activities one hour per week for eight weeks. 

• Two video-specialists from Columbia Center for New Media Teaching and Learning (CCNMTL) 
edited the video, linked files, requested captioning, and uploaded video, for three to five hours 
per week totaling around 32 hours. 

• A senior educational technologist from CCNMTL served as the day-to-day project manager for 
the MOOC production and delivery, spending eight hours per week for four weeks prior to 
launch, five to six hours per week over the first few weeks of course delivery, two to three hours 
per week over the last few weeks of course delivery, and ten hours after the MOOC finished, for 
a total of around 75 hours. This included monitoring the online discussion forum for technical 
questions, averaging three hours per week over eight weeks for a total of 24 hours. 

Figure 5: Faculty Member's Hours by Task for Development and Delivery of Big Data in Education 



Hours 
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Cost analysis of Ryan Baker's MOOC: Big Data in Education 

Based on the above information, we estimated the personnel costs required to replicate the 
development and delivery of Big Data in Education using national average salaries and benefit rates for 
postsecondary personnel at private universities. We obtain an estimate for personnel costs of $29,240 
(see Table 10). We have not estimated the costs of facilities, computers and other equipment, or 
overhead, but if we assume that personnel costs account for 75% of the total costs for the course (Levin 
& McEwan, 2011), the total estimate would be $38,980. The costs of the platform, captioning, content 
hosting, and analysis of user data to populate the data dashboard were assumed by Coursera. 

Table 10: Personnel Costs for Big Data in Education Development and Delivery 


Personnel Ingredients 

Cost 

Faculty member 

$12,354 

Teaching assistant 

$4,950 

Forum monitors 

$5,377 

Education administrator - educational support services 

$584 

Educational technologist: project manager 

$4,725 

Video Editor 

$1,248 

Total Personnel Costs 

$29,238 


In a sensitivity analysis, we substituted the national average faculty member salary (which averages 
across assistant, associate, and full professors) for the average full professor's salary at a research 
institution. This yielded a total cost estimate of $45,010, compared with our baseline estimate of 
$38,980. In a second sensitivity analysis, we used the average salary for a full professor at Columbia 
University ( http://chronicle.com/article/2013-AAUP-Facultv-Salarv/138291?cid=megamenu) and 
adjusted prices to reflect a New York metropolitan area. Resulting costs totaled $69,510. In a final 
sensitivity analysis, we substituted the estimated costs of externally contracted video production 
($4,300 per hour of finished video as shown in Appendix IV ) for the video editors. This substantially 
increases total costs to $76,250 if we use the national average cost of a faculty member, and to 
$110,950 if we use the average salary and benefits of a full Columbia University professor and New York 
prices. A summary of these sensitivity analyses is shown in Table 11. If Big Data in Education were run 
multiple times using the same materials, the costs of time invested in producing the course, which we 
estimate to be 70-80% of the total, could be spread over the multiple runs. 

With 526 students completing Big Data in Education, estimated costs per completer using the baseline 
cost analysis would be $74. If the course were developed and delivered by a full Columbia University 
professor and an external video production team, with prices adjusted for a New York metropolitan 
area, costs would rise to $211 per completer. 
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Table 11: Estimated Replication Costs of Big Data in Education under Various Assumptions 


Faculty and benefits 
assumptions 

Geographical 

adjustment 

Video 

Production 

Total estimated 
costs 

Estimated costs 
per completer 

National average private 
university faculty 

N/A 

Internal 

$38,980 

$74 

National average private 
university full professor 

N/A 

Internal 

$45,010 

$86 

Average full Columbia University 
professor 

NY, 

metropolitan 

Internal 

$69,510 

$132 

National average private 
university faculty 

N/A 

External 

$76,250 

$145 

Average full Columbia University 
professor 

NY, 

metropolitan 

External 

$110,950 

$211 


Non-quantified consequences of MOOC production 

As a result of spending time on the development and delivery of this MOOC, Baker documented 
significantly lower productivity compared to the prior year in terms of submitted conference proposals, 
journal articles, and book chapters (see Table 12). It is not obvious how to quantify these changes in 
economic terms but there is clearly a trade-off to be considered in terms of the professional status and 
institutional visibility to be gained by delivering a MOOC to thousands of diverse participants vs. 
disseminating research findings to an audience of conference attendees and journal or book readers. 
While the number of grant proposals that Baker submitted remained unchanged, it is possible that 
acceptance could be greater or lower depending on the funders' perceptions of the value of one activity 
over another. 

Possible benefits to both the instructor and the institution could accrue going forward if grant proposals 
were more likely to be accepted due to a MOOC being seen as a viable vehicle for the large-scale 
dissemination of research findings. Additionally, delivery of the MOOC could result in a higher number 
of applicants to Teachers College programs in general and to the M.A. program in Cognitive Studies in 
particular. To track this it would be necessary to include a survey item in the admissions package asking 
what factors had influenced the applicant's decision to apply, with the MOOC being offered as one of a 
number of possible options. 


Table 12: Professional Productivity Changes as a Result of Time Devoted to Big Data in Education 


Items submitted 

Sep '12 -Feb '13 

Sep '13 -Feb '14 

Year-to-year change 

Research grant proposals 

6 

6 

None 

Journal articles/revisions 

24 

13 

-46% 

Book chapters 

9 

3 

- 67% 

Conference paper proposals 

18 

12 

- 33% 


Sources: Ryan Baker, Teachers College, Columbia University, NY; Yuan "Elle" Wang, Teachers College, Columbia 
University, NY; Michael Cennamo, CCNMTL, Columbia University, NY. 
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Where are MOOCs Going over the Next Five Years? 


The final two open-ended questions of our interview protocol invited interviewees to predict how the 
MOOC phenomenon might play out over the next five years: 

• Where do you see you see things going with MOOCs over the next five years? (Generally, and at 
your institution). 

• How do you see MOOCs helping your institution meet its goals over the next five years? 

Most interviewees were willing to speculate about how MOOCs might evolve over the next few years 
and how they might fit into the changing landscape of higher education. We begin this section by 
summarizing expectations about more general changes in higher education and subsequently present 
specific ideas raised regarding MOOCs of the future. Our goal was to surface a variety of ideas as 
opposed to developing a rigorous frequency report on a list of possibilities from a stratified sample of 
participants. Accordingly, we provide some idea of the "popularity" of each idea by loosely indicating 
how many interviewees mentioned it, using terms such as "many, some, several, a few, a couple, one." 
Given the open-ended nature of the questions and variety of interviewee roles, we caution readers 
against focusing too much on this quantification. 

The future of higher education 

Many interviewees envisaged a future of more "unbundling" of educational services, more choice for 
students in how their education is delivered, greater price competition among providers, and greater 
use of technology in education. As BenDaniel from Cornell University observed "University education is 
reaching a point of diminishing returns as the cost of university staffing is growing unsustainably.... 
something will change." Several questioned whether two- or four-year degrees are the most useful way 
to acquire an education and suggested that while colleges and universities may continue to provide 
foundational skills and knowledge, the flexibility of online education will allow for more "just-in-time" 
learning experiences throughout an individual's career. Several interviewees subscribed to the notion, 
often attributed to President Rafael Reif at MIT, that four-year residential education might someday be 
replaced with three years of residential courses and one year of online learning. Overall, the need for 
flexibility of educational options was often repeated, as typified by Royal at Inver Hills Community 
College: "We should have a series of different educational environments that suit the diversity of our 
student base." 

Most interviewees believed that higher education will be increasingly pressured to shift towards 
competency-based models of credentialing. Several foresaw the possibility that the emphasis on earning 
a degree will dwindle in favor of an emphasis on demonstrating capabilities needed by employers. As 
Kirschner at Macaulay Honors College asserted, the burden of showing what a student can do will "shift 
away from the degree as a proxy" because for some employers, higher education has "failed to make 
the degree stand for something." Concomitantly, Cator of Digital Promise favors a shift in federal funding 
for postsecondary education away from simply paying for access and towards paying for the 
demonstration of competency and attainment of credentials: "the incentives then would be all around 
performing, producing, learning, rather than around getting into a class, going to college, or not going." 
Smith, of Straighterline, carries this vision even further to a future in which federal funding for college is 
replaced by life-long education accounts that individuals can use to select their own educational 
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experiences, provided each one meets minimum standards of accreditation (see Smith, 2013). Minghua 
Li, Professor, School of Public Administration at East China Normal University, believes that the major 
online platform providers should be collaborating to create a course accreditation system and could 
subsequently pressure the federal government into allowing the application of financial aid to pay for 
online courses offered outside of college degrees, including MOOCs. 

A few interviewees conjectured that, as more education is offered online, some institutions of higher 
education could shrink and disappear while virtual universities, or virtual extensions of established 
institutions, expand. However, many interviewees doubted that MOOCs would dramatically affect brick- 
and-mortar colleges and universities. Most interviewees expected to see changes in pedagogical 
strategies, for example, more flipped classrooms and greater emphasis on peer learning. Increased use 
of both gamification and game-based learning strategies is expected in some content areas to "hook" 
students through constant feedback, challenge, and creation of a social community. For some 
interviewees, the social community aspect is one of the most powerful that can be developed in online 
environments. Parker of NextThought envisions "tech-enabled learning communities that will persist 
beyond the university experience providing individuals with a life-long network of peers across time and 
space." 

One concern raised by Newfield at the University of California, Santa Barbara, among others, is that “the 
genie of free, quality education is out of the bottle", that is, the expectation that high quality educational 
experiences can be delivered at negligible cost has caught on, despite the lack of evidence to show 
whether MOOCs and similar experiences offered at scale can substitute some part of face-to-face 
education. Newfield worries that well-intentioned politicians and business leaders will continue to press 
for degrees costing $10,000 or less, and will fail to invest enough in public education. Silverman at NYU 
fears that to combat escalating costs, institutions will ever more aggressively replace tenured professors 
with talented teachers to deliver the curriculum, but these instructors will not participate in the 
continuous research necessary to constantly push academic disciplines forward. 

MOOCs and the future 

We categorized the assortment of ideas offered by interviewees regarding MOOCs in the future of 
education into three general topic areas: 1) those that describe how the courses themselves might 
evolve from current MOOCs; 2) ideas that capture how MOOCs may affect aspects of the current 
economic model of higher education; and 3) ideas about how MOOCs can facilitate relationships among 
institutions of higher education, and between these institutions and other parties. We summarize the 
ideas in Table 13 and loosely indicate the frequency with which each idea arose. The remainder of this 
section provides additional detail on each idea. 
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Table 13: Ideas Raised by Interviewees about MOOCs in the Future of Education 


Ideas Raised by Interviewees about MOOCs in the Future of Education 

Mentions 
among 83 
interviewees 

How MOOCs will evolve as courses 

MOOCs will serve as educational resources rather than as stand-alone courses 

Many 

MOOCs will be targeted at specific audiences 

Many 

MOOCs will offer revenue-generating services such as tutoring, face-to-face 
interactions, and study groups 

Many 

MOOCs will offer credentials of economic value such as high school or college credits, 
or badges that are recognized by employers 

Some 

Learning analytics data will catalyze the advent of personalized, adaptive, and mastery- 
based learning 

A few 

Convergence of cMOOC and xMOOC pedagogy 

A few 

Continuous A/B testing will allow for iterative improvements in materials and activities 

A few 

MOOCs will incorporate computer grading of open-ended assignments 

A few 

MOOCs will be available on mobile devices 

A couple 

How MOOCs could affect the economic model of higher education 

Students will cobble together their own certifications 

Many 

Meeting demand for higher education in rapidly developing countries 

A few 

Faculty members as free agents 

A couple 

MOOCs-lnside platform and course markets 

A couple 

Replacing the course catalog and course "shopping" 

One 

How MOOCs can facilitate relationships among institutions 

Cross-institutional collaborations 

Many 

Life-long connections to alumni 

A few 

Community outreach 

A couple 
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I. How MOOCs will Evolve as Courses 

Starting with Koh at Coursera, all interviewees expected that MOOCs and MOOC platforms, assuming 
they still exist, will look vastly different and more sophisticated in five years' time. While not everyone 
was certain that MOOCs would persist, many expected them to endure either as stand-alone learning 
experiences or as the online portion of the increasingly popular flipped classroom model. MOOCs were 
expected to evolve in a number of ways and more than one cynical interviewee observed that this 
evolution could lead MOOCs to more closely resemble regular online courses of today. 

MOOCs will serve as educational resources rather than as stand-alone courses 

Many interviewees expressed a sentiment similar to that of Dodds at Cornell University that MOOCs will 
"evolve into high-end, multi-media textbook replacements." Lester at FutureLearn envisions the 
possibility of a tagging system within MOOC materials to allow students and faculty members to search 
for relevant items. 

We could tag things within courses. That would probably take quite a bit of work, but if it was 
planned up front so it would just build up, then it could be done. In which case, you open up the 
fact that MOOCs are being used as resources, much as a textbook would, rather than as a 
course. (Lester, FutureLearn). 

MOOCs will be targeted at specific audiences 

Many interviewees perceived that a weakness of current MOOCs is their need to present content and 
activities that are accessible to participants with a wide range of incoming knowledge and skills. These 
interviewees generally felt that MOOC pedagogy and completion rates could be improved by targeting 
specific audiences: what Silverman at NYU terms "narrowcasting" in contrast to the "broadcasting" he 
associates with most MOOCs. While this may reduce the number of participants, it is more likely that 
those who enroll will be active in the course. Examples of such targeting included MOOCs for K-12 
students, MOOCs for teacher professional development, and MOOCs for corporate training. 

MOOCs in K-12: Kinzler from AMNH believes that "There is a lot of potential for digital content to be the 
direct instruction component for K-12," thereby allowing teachers to "...be more facilitators of meaning- 
making than deliverers of content." A few participants suggested that MOOCs might help high school 
students who do not have access to teachers in less common subjects such as chemistry, physics, or a 
foreign language. However, as noted earlier in this report, the prospect of MOOCs in K-12 raises a 
number of policy challenges. For example, public funds are currently apportioned to schools on a per- 
student basis and do not follow a student taking courses offered by other institutions. To avoid some of 
the issues endemic to the K-12 regulatory environment, Paska at Harrison Central School District 
suggested that MOOCs could provide credit towards an Advanced Placement (AP) degree or towards 
college. He speculated that Boards of Cooperative Educational Services (BOCES), or similar organizations 
in other states, might be able to facilitate the introduction of MOOCs by using existing Cooperative 
Service agreements (CoSers) to share courses across district lines, and by soliciting entrepreneurial aid 
to support MOOC development. 

MOOCs for teacher professional development: Kleiman at North Carolina State University reports a high 
degree of interest from state-level officials regarding the possibility of using MOOCs, perhaps in a 
blended model of learning, to meet the professional development needs of teachers and administrators. 
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This audience could well be expected to fit the mold of well-educated, motivated learners who might 
fare well in MOOCs, particularly if completion of a course is a job requirement and professional time is 
dedicated to it. 

MOOCs for corporate training : A number of interviewees observed that corporations have taken a fair 
amount of interest in MOOCs. Rizvi at Pearson alluded to a McKinsey survey -finding that many European 
employers find university graduates ill-equipped with the skills required for entry-level jobs and he sees 
a role for MOOCs in closing this gap. He suggested that traditional universities are not necessarily the 
best source of training on certain topics, for example, Google could provide MOOCs on programming 
and L'Oreal on marketing. Already, Rizvi noted, "the big industrialist families” in Pakistan and the 
Philippines that employ hundreds of thousands of people are establishing their own universities to help 
prepare graduates with the skills needed by these employers. Lester at FutureLearn similarly pointed 
out that MOOCs may be used to address some of the tensions that currently exist between business and 
universities: 

Universities have an interest in guality, governance, and reputation. It takes a very long time to 
produce new content while business and technology move very fast. Businesses want immediate 
responses; turnaround of content that is relevant and useful. Universities traditionally haven't 
been very good at serving those markets. There is a market failure there. (Lester, FutureLearn). 

Lester noted that Google and Apple have already built their own course platforms and he pointed to the 
MOOCs created by Enterasys Networks as a potential model for technical training in the future. The 
possibility that MOOCs will be used for corporate training purposes was also raised by other 
interviewees. Many pointed to the announcement in late 2013 by Udacity of a shift away from working 
with institutions of higher education in favor of corporations. Three of our interviewees from platform 
providers reported ongoing discussions with large corporations and, unusually, we encountered 
representatives of several corporations at the MOOC Research Initiative in December who attended to 
learn about MOOCs and explore their applicability for internal use, or for collaborative initiatives with 
institutions of higher education. Saxberg and Niemi at Kaplan Inc. described plans to develop a MOOC 
on cognitive task analysis. Beyond our own study sample, Radford et at. (2014) report that while only 7% 
of employers they surveyed in North Carolina had already used MOOCs for employee professional 
development, another 76% had either considered doing so or would be open to the idea. 

MOOCs will offer revenue-generating services such as tutoring, face-to-face interactions, and 
study groups 

There was consensus among our interviewees that MOOCs will not be indefinitely sustainable without 
some form of revenue or demonstrable impact on reducing the costs of education. Interviewees 
expected revenue-generating services to be offered to MOOC participants, such as online tutoring, 
opportunities for face-to-face interactions with a local instructor and a local cohort of participants, or 
placement in an online study group with a few carefully selected peers. 

Many interviewees believed that maximum benefit will be derived from MOOCs only if opportunities are 
provided for participants to work simultaneously face-to-face with an instructor. While localized meet- 
ups have been organized by the platform providers and institutions creating MOOCs, these interactions 
are only likely to be effective at improving the educational impact of MOOCs if students can follow up 
with some sustained educational activity, for example, establishing small, collaborative study groups. 
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If I had to launch a MOOC today I would look for more ways to enhance the level of engagement 
of the course and look to create some distributed but localized gatherings like student meet-ups 
so the students could have a connection to rely on, encourage, and motivate them. There are 
very subtle ways in which group dynamics contribute to you feeling a sense of desire to continue 
and get things done because you are pacing with the rest of the class. (Royal, Inver Hills 
Community College). 

MOOCs will offer credentials of economic value such as high school or college credits, or 
badges that are recognized by employers 

Some interviewees expressed the view that the MOOC movement would soon fade out unless 
participants are able to earn meaningful credentials carrying economic value. Miles at Tri-C anticipates 
that, over the next few years, students will ask for credits for MOOCs completed and the college will 
have to figure out how to award credit based on competency. 

MOOCs may offer traditionally accepted credentials such as high school or college credits, and 
professional certifications, or newer types of recognition such as virtual badges. While a small number of 
MOOC participants are already paying for verified certificates of completion, employers are likely to 
want confirmation that job candidates can actually apply skills and knowledge in practice, perhaps even 
using secured online environments to test these during the job interview process. 

The concept of badging ... provides an opportunity to link business and industry. I think both 
business and educational institutions have concluded that there is a need for better connections 
between employer expectations and student preparation, and, to me, badging could be another 
linkage to help reduce the mismatch of expectations and needs. So if an employer can see who 
has badges in specific competencies that the employer is seeking, that would be helpful. (Royal, 
Inver Hills Community College). 

The whole issue ofcredentialing is going to be critical in this. ..The real test of whatever system 
we are going to have for credentialing is what the employers who hire the individuals with the 
certification take as markers for people that are adequately prepared to contribute value to their 
businesses. That is going to be the measure. Who knows, it may be that large corporations say 
that students trained under the new models are equally as productive as the traditional models, 
then I can imagine that we are looking at real, substantive transformation of higher education. 
(Professor, East Coast University). 

Radford etal. (2014) report that while only 31% of 103 employers surveyed in North Carolina had even 
heard of MOOCs and only one had used a MOOC provider's recommendation for recruitment purposes, 
73% were favorably inclined towards considering completion of MOOCs in hiring decisions: "...while 
traditional degrees and credentials were still necessary qualifications for jobs, respondents tended to 
view MOOC course taking as an indication of greater motivation" (p.2). This finding bodes well for the 
future of MOOCs. 

Learning analytics data will catalyze the advent of personalized, adaptive, and mastery- 
based learning 

A few interviewees expect that truly adaptive learning will indeed materialize over the next few years so 
that the MOOC experience can be tailored to each individual participant. Laster at McGraw-Hill 
Education observes that adaptive learning solutions are not simple or inexpensive to develop as each 
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course must be developed with the adaptive learning strategies built in. This may require working with a 
third-party provider of adaptive learning solutions or developing the capacity in-house. In either event, 
the process requires an investment of time from subject-matter experts, instructional designers, 
instructional technologists, data analysts, and computer programmers. As these constituencies are 
already being represented in many MOOC production teams, it is likely that some of these teams will 
progress towards tackling the adaptive learning challenge over the next few years. 

To provide learning on an adaptive and personalized level, the capacity for MOOC platforms to track and 
report individual level data must first be increased. As Whitmer suggested "the next frontier will be 
looking for patterns in learner behavior over time and over multiple courses , and looking for relationships 
between activity and measures of achievement." Renick, who is well versed with the use of student data 
and predictive analytics at Georgia State University, believes that MOOCs could be helpful in the 
university's ongoing quest to improve course completion and graduation rates "if they can provide really 
granular understanding of how students are performing long before final grades are submitted." 

Convergence of cMOOC and xMOOC pedagogy 

From a pedagogical perspective, MOOCs of the future are expected to converge between the current 
extremes of connectivist MOOCs, which encourage creation of artifacts and networked learning, and 
xMOOCs which are more about transmission of knowledge and what educators might term "drill and 
practice." Several interviewees anticipated that course designers will adopt some combination of the 
social aspects of cMOOCs and the structured presentation of content and activities typical of xMOOCs. A 
key aspect of this shift will involve platform design that facilitates peer learning and enables more 
efficient collaborative work. 

As reported under Goal 6: Conducting Research on Teaching and Learning, Kulkarni and others at 
Stanford are already investigating the optimal composition of study groups. A small number of 
interviewees envisioned the capability for MOOC platforms to identify student characteristics and to 
recommend productive combinations of students for online study groups. Dede at Harvard anticipates 
delivering a "massive learning experience" that incorporates "an automated pattern-matching tool that 
creates teams that have complementary knowledge and skill, based on the surveys students take before 
the course." The objective of the course "on peer learning and by peer learning" will be to help 
participants learn how to build social capital and develop other non-cognitive skills such as 
collaboration, metacognition, and tenacity. Ideally, the experience will render participants more 
effective at peer learning so that they can maximize the benefits from future participation in learning 
experiences such as MOOCs. Insights obtained from this larger audience may be adapted to improve 
peer-learning on-campus. Parker similarly expects future versions of the NextThought platform to allow 
formation of small groups guided by student attributes. Experience and research will help identify the 
best predictors of individual and group success. 

Continuous A/B testing will allow for iterative improvements in materials and activities 

While the Coursera platform currently facilitates A/B testing (see Box 5) by allowing delivery of different 
e-mails to participants at random, several researchers we interviewed did not expect it would be long 
before A/B testing will be possible directly within the Coursera platform and others. This would allow 
different materials or activities to be presented to different students, and an investigation of whether 
these variations affect student behavior and performance. If it appears that a particular variation has led 
to an improvement in participant persistence or performance, future iterations of the course could 
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incorporate the relevant feature. This presumes that there is both funding available and institutional will 
to keep perfecting a MOOC once it has been developed. 

Lester of FutureLearn also envisions the use of A/B testing to fine-tune instructional materials, but he 
acknowledges that unless it can be demonstrated that learning outcomes have been improved, such 
efforts would be in vain. 

We will do A/B testing and figure out, "This video, or videos with a human face on it, actually do 
far better than the videos without a human face," and so on. We will constantly learn and find 
out what the students want to do, and what they engage best with. (Lester, FutureLearn). 

MOOCs will incorporate computer grading of open-ended assignments 

While much of the current focus with respect to grading of MOOC assignments is on improving peer- 
grading strategies, a few interviewees commented on expected developments to improve the capacity 
for computer-based grading of open-ended assignments such as essays. Depending on the instructor's 
subject area, this development was more welcome or less so. Some strongly argued that computers are 
incapable of capturing the nuances an experienced instructor would perceive, while others, including 
Baker at Teachers College, were more sanguine. 

MOOCs will be available on mobile devices 

Dillenbourg et ai, (Eds., 2014) describe ways in which engagement in MOOCs via mobile devices can 
improve "information access, production and collection of data by learners, and communication." 

Mobile access to online course content is already available to some extent from some providers such as 
2U (see Press Release) and Coursera. NextThought allows for access to MOOC course content via iPad 
tablets (see Case 8 ) and offers a mobile notification strategy such that if a participant initiates a thread 
or asks a question, he or she immediately receives an e-mail or instant message when a response is 
posted. A single click takes the participant back to the online dialogue. CEO Parker is confident that 
mobile devices will increasingly supplant paper and even laptops, and he expects that a fully interactive 
MOOC experience will soon be supported at least on Android tablets and smartphones. It is likely that 
other platform providers will follow suit. 

Mobile allows for omnipresence and immediacy. Dialogues fade out when too much time elapses 
between the interactions, that's why the social networking sites have invested hugely in mobile 
strategies. Mobile devices will play a key role in connecting people. (Parker, NextThought). 
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II. How MOOCs could Affect the Economic Model of Higher Education 


While interviewees considered it highly unlikely that MOOCs will provide a complete cure for higher 
education's "cost disease" (see Bowen, 2012), many identified ways in which MOOCs may affect the 
current economic model of higher education, both in the U.S. and abroad. Several interviewees 
speculated that campus-level versions of commonly offered introductory courses such as Statistics 101 
or Economics 101 could be each replaced by a single "master" course, in some cases a MOOC, that could 
be supported locally by non-tenured instructors or teaching assistants. Kalbaugh at Wake Technical 
Community College believes that MOOCs may more cost-effectively serve the role previously played by 
community colleges of providing low-cost, low-risk, and life-long learning opportunities to non-degree 
seeking adults. But Kalbaugh, and several other community college administrators, do not see MOOCs 
serving their core, degree-seeking student body more effectively than current course offerings. 

Students will cobble together their own certifications 

Perhaps the most common theme we heard regarding the future of MOOCs is the role they would play 
as one of many choices available to students who will increasingly be able to pick and choose their own 
educational experiences, which may or may not include a traditional college degree. Fisher at Vanderbilt 
University observed that it is already possible to take all the necessary courses for a computer science 
degree through MOOCs (see Fisher & Fox, 2014, Figure 1, p.4), with only the lack of credits preventing 
anyone from actually earning a degree this way. He foresees the possibility that students will be able to 
design their own majors in MOOCspace, provided there is some means of accreditation. 

Kirschner at Macaulay Honors College predicts that the burden of responsibility for demonstrating a 
person's capabilities will shift away from educating institutions to the students themselves: 

...this is part of a larger shift towards self-credentialing where the student is in control of saying 
" This is what I know how to do." For example, "I can do statistics or analyze your data and here 
is my evidence." The student is going to have to aggregate learning experiences of certain types. 
Some of these will be certified, or the students will assemble an e-portfolio that stands as 
evidence of what they can do. I don't think the one-size-fits-all B.A. credential will stand as the 
dominant form of credentialing. 

Where the revolution is going to come first is with the smartest, best-prepared kids, not 
the least-prepared students. There's always going to be 10,000 applicants lined up to go to 
Harvard, but will they continue to be the smartest 10,000 applicants? More and more of these 
students may say: "I had a really good high school education and I'd like the partying, hanging 
out, and so on, but my family can't afford it. So do I need a four-year degree at enormous cost, 
or should I take one year at college, one year studying abroad, and fill in with MOOCs and 
internships?" (Kirschner, CUNY). 

To facilitate such a change, Smith at Straighterline believes that students should be able to choose from 
a menu of options and apply public funding to pay for their choices. 

I would look at all the subsidies that higher ed enjoys from the federal government, directly in 
terms of loan subsidies, or tax credits, or direct state support, or nonprofit status, all these 
different ways that higher ed is subsidized. They all add up. There are half a dozen subsidies, so 
it's a pretty enormous number. Take that, and set some minimum baseline standards, and 
allocate the rest in some form of educational, or training, or life-time educational account that 
people can access once they turn, say, 18. You can adjust it for different socioeconomic 
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categories. Let people make their own training decisions with those dollars. That's the kind of 
model I'd like to see. ..something more like vouchers or food stamps, as opposed to this kind of 
patchwork subsidy structure that we currently have. (Smith, Straighterline). 

Straighterline already offers what may become a more popular model of higher education in the future. 
Students can sign up for $99 per month to take as many or as few courses as they wish. Services include 
online tutoring provided by a Pearson-owned company, Smart Thinking; student advising including 
technical support; proctoring services from Proctor U; and free transcripting. All Straighterline courses 
carry ACE credit recommendations. In addition, Straighterline has negotiated formal articulation 
agreements with 65 institutions to guarantee credit for Straighterline courses. Smith notes that an 
equity issue may arise because the most motivated students require the least amount of support, and 
are therefore the cheapest to serve, while less motivated students need the most support services. If 
stronger students do not pay for services they are not using, there is no subsidy for the weaker students 
who do need these services. The weaker students could end up paying more to earn the same 
credential, providing yet another disincentive to education. 

Downes at the National Research Council is working towards the development of advanced personal 
learning environments (PLEs) through the Learning and Performance Support System project. The goal is 
to allow each individual learner to set up his or her own network of resources independently of any one 
provider. Each learner's PLE would link to a variety of learning resources offered by multiple providers, 
including MOOC platform providers, corporate trainers, commercial education companies, free-lance 
instructors, and traditional institutions of higher education. 

Not everyone was as enamored of this idea, and some expressed doubt that the majority of students 
could be self-directed enough to take advantage of such freedom of choice. For example, Paska noted: 

One problem I see with the possibility of earning a MOOC degree is how do I know what I don't 
know? Students don't always know what they need and often they need a seguence of 
instruction to obtain the requisite knowledge and skills. (Paska, Harrison Central School District). 

This, of course, leaves the door open to online advisory services that can help guide students through a 
maze of online learning opportunities to construct a coherent set of credentials. Kirschner at Macaulay 
Honors College believes that there is much to be done to help learners navigate the rapidly growing 
choices: 


Who will be the Amazon of online learning? If students are going to be cobbling a little bit from 
here and from there, who is going to rank, for instance, the available stats courses? If I want a 
job at a pharmaceutical company and am interested in stats, I should be able to go to 
somewhere and see the training and courses needed for this job and, of all online stats courses, 
some designation that these are the ones that most people in those jobs say are the best. There 
are companies already looking into this, like Balloon, Skilled Up, and Lynda.com. That's the lens 
into education that I think is so revolutionary. (Kirschner, Macaulay Honors College). 
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Meeting demand for higher education in rapidly developing countries 


A few interviewees expect that MOOCs could help address the shortage of universities and faculty 
members in developing countries. Peercy at the University of Wisconsin-Madison observed a drastic 
shortfall in qualified faculty members abroad to meet the rising demand for engineering degrees and 
believes that MOOCs could play a significant role in addressing this concern. Similarly, Mintz at the 
University of Texas System envisions specially developed MOOCs to fulfill the unmet need for higher 
education in developing countries. 

In the developing countries they cannot create universities fast enough to meet the need. They 
face all kinds of challenges. In India, currently ten percent of college-aged students enter college. 
Clearly that needs to rise, but how can they do that in an affordable manner? The answer seems 
to me new kinds of MOOCs, or some kind of hybrid delivery using MOOCs. I'm hoping that edX 
will move into that area. Not targeting high-end courses to autodidacts and degree holders, but 
standard undergraduate classes. 

They're going to have to develop a model to incentivize campuses to create those 
courses for free. Probably what will have to happen is some foundation will have to pay for those 
courses. They would have to be developed in some collaborative fashion with the country that 
would consume them. It would also have to be in the indigenous language. It may not be done by 
the most famous faculty members, and it may be done through the Education College or the 
Teachers College. What I envision is edX identifying courses that need to be produced, running a 
competition in which everybody would be invited to participate, and then choosing the best 
development team that it could find that's most likely to produce what we really want. (Mintz, 
University of Texas System) 

One of the most interesting proposals we heard for using MOOCs to provide higher education in 
developing countries was University of Colorado Anschutz Medical Campus's idea to offer a MOOC for 
global health responders that, for students who complete the MOOC successfully, would be 
supplemented with a two-day, face-to-face, hands-on certification course, designed by University of 
Colorado faculty members and delivered at locations around the globe. 

Faculty members as free agents 

A professor at an East Coast university described a future in which the massive data from MOOCs could 
be used to identify particularly effective teachers who might then act more like free agents: 

Let me just describe a possible path of evolution. I think the final end point is faculty free agency. 
As I think about what the technology might be able to do, the model I tell people is to think 
about baseball and free agency. The reason it is relevant is there is a reality that some teachers 
are just better teachers than others. 

Because there is little in the way of data by which one can compare, there is little 
commonly agreed upon science around this. [With MOOC data] you might be able to identify 
first a set of individuals who are far more effective at getting learners to a high level of 
competency in a given time, the equivalent of sports superstars. Once you identify those 
individuals, we can work on what is it that they are doing in their classrooms that makes them so 
effective. Then we can train other people to do it. 

It is completely possible that at some point these faculty members will not work for a 
university. May be some people are so good at this that a commercial entity might say that 
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instead of working for the university, come work for us. ..Using the power of the market to 
identify extraordinarily successful individuals is exciting. (Professor, East Coast University). 

Parts of this vision are already appearing at Straighterline, where qualified instructors from any 
institution can apply to provide supplementary and support services to students enrolled in 
Straighterline's online courses. According to Straighterline's CEO, Smith, these faculty members can earn 
$50-$100 per student for providing these extra services. While there may be limits to how many 
students any one instructor can effectively support, if faculty members could earn royalties based on the 
number of students they attract, there may well be outsize earnings accruing to master teachers in the 
future. 

MOOCs-Inside platform and course markets 

Minghua Li of East China Normal University envisions a future which encompasses a number of ideas 
both about how MOOCs will change and how higher education will change. He expects that, in the 
future, mainstream higher education courses will be hybrids of MOOCs with local face-to-face 
interactions in a classroom. He proposes the idea of a "MOOCs-Inside platform" that will facilitate the 
development of an accredited independent course market operating in parallel to the current "degree 
market." Instructors will not need to be affiliated with a university, but only with one or more platforms 
through which he or she will offer courses and be connected to students. Minghua Li foresees that some 
institutions will primarily produce MOOCs while other will "consume" them. Some institutions will 
attract fewer residential students and can rent out underutilized facilities to local instructors hired by 
the MOOC-producing institutions, or to free-lance instructors associated with the platform providers. 
Reynolds from Technology Crossover Ventures similarly predicts a future of course markets rather than 
degree markets, comparing the shift with that of music from CDs to iTunes. 

Replacing the course catalog and course "shopping” 

Sreenivasan, formerly Chief Digital Officer and Professor at Columbia University, noted the positive 
impact of Coursera's requirement that MOOC materials be ready at least two weeks in advance of 
delivery. He believes that the customary, but fraught, shopping period at the start of each semester can 
be replaced by a course catalog composed of MOOC-style introductory videos to help students preview 
and select courses. 

My goal is to have 3,200 one-hour previews or trailers of each of our courses. The students live in 
a preview culture: before they go to a movie they see a preview, before they buy a book they 
read a chapter. Why is this only thing in their lives where they don't have a preview? 
(Sreenivasan, formerly at Columbia University). 
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III. How MOOCs can Facilitate Relationships among Institutions 


MOOCs appear to have presented numerous opportunities for colleges and universities to engage with 
outside entities, including other institutions of higher education, corporations, funding agencies, local 
and distant communities, and alumni. Many interviewees expected more cross-institutional 
collaboration to share resources and expertise, and to spread costs that could not be borne by a single 
institution. 

Cross-institutional collaborations 

A number of cross-institutional collaborations have already been formed to offer online courses, 
including MOOCs, and several interviewees expect to see this trend grow. Griffiths at Ithaka S+R pointed 
to the Semester Online initiative facilitated by 2U and suggested that similar consortia could soon offer 
MOOCs that would be accepted for credit by partner institutions. However, the ephemeral life of 
Semester Online demonstrated that such collaborations among institutions of higher education are not 
always easy to negotiate and sustain ( Straumsheim, 2014) . 

According to another interviewee, some of the fifteen Committee on Institutional Cooperation (CIC) 
state universities are considering extending their "coopetition" arrangement that originated with 
sharing computer hardware and other resources, to pool funds for the development of MOOCs. 
Nominations would be submitted to the fund for MOOC ideas and some of these MOOCs could be 
jointly developed across a few universities in the consortium. The goal would be to attract applicants to 
full courses or programs offered by the universities. Our interviewee suggested that these MOOCs could 
be integrated with admissions procedures, for example, by encouraging top performers to apply to 
programs offered by CIC universities. He also suggested that reviewing participant persistence in one or 
more MOOCs could help in making financial aid decisions to help avoid the losses incurred when aid is 
awarded to students who subsequently drop out. 

Mintz at the University of Texas System highlighted the example of an initiative organized by HASTAC 
which involved Cathy Davidson's MOOC History and Future of (Mostly) Higher Education being offered 
concurrently with on-campus courses led by Davidson at Duke University, Palumbo-Liu at Stanford and 
Newfield at the University of California, Santa Barbara. 

You have a face-to-face intimate encounter taking place within a context of global connectivity 
and featuring the biggest names from around the world who've thought about the future of 
higher ed. It's going to be through Coursera, and HASTAC is the organizer. To me this is 
tremendously exciting, and a potential model for really interesting next-generation cross- 
institutional collaborations. (Mintz, University of Texas System). 

Fisher at Vanderbilt University is excited about the possibilities for what he calls "crowd-sourced 
curriculum" whereby one institution develops a MOOC and another follows it with a complementary, 
sequenced offering: 

We are going to see more collaborative agreements between MOOC providers and more 
interactions between MOOC courses. For example, we are currently working on a seguence. 
University of Maryland will offer a MOOC and we will follow that up with another MOOC. It is 
not quite a prerequisite, but if you want to take two sequenced MOOCs, they are coordinated 
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that way. We are going to see more of that. We are going to see various kinds of interactions 
between MOOCs. (Fisher, Vanderbilt University). 

In late 2013, the State of California announced a $57 million Online Education Initiative to develop 
online courses that could be used across all 112 community colleges in the state (see Raths, 2014 ). The 
goal of the initiative is to increase the number of students who earn a two-year college degree and 
subsequently transfer to four-year colleges. While it is not clear how closely any of these courses will 
resemble MOOCs, it is likely that shared courses will need to serve very large numbers of students. One 
of our interviewees described the potential for this initiative: 

...we can imagine that all 112 community colleges in California work together to orchestrate a 
consistent course management system and course schedule portal where any community college 
student can enroll in any of the available courses developed by different community colleges, but 
they would have a consistent learning experience. The coordinated effort would allow provision 
of 24/7 technology support and online tutors and advisors, which none of us could afford 
individually, and these courses would all be open access and available for credit. (Online learning 
coordinator, Large California Community College). 

Royal, at Inver Hills Community College, cautions that collaborations are never easy and are often born 
of necessity. 

...partnerships take a lot of work, there are differing beliefs about what the outcomes should be 
for the course and therefore what the lessons should be, and faculty don't always like to give up 
their autonomy. It is hard to negotiate expectations. But I think more institutions will be forced 
to partner as financial resources become scarcer. For example, if someone wants to do a MOOC 
but doesn't have enough money, either they partner or they cannot be in the game. When 
people don't have the money to do things, that makes them creative. Money can be a driver into 
partnerships. (Royal, Inver Hills Community College). 

Collaborations are also likely to form around efforts to develop online platforms in addition to existing 
consortia such as edX. Collier at Stanford University anticipates that the need for third-party platform 
providers to generate revenues will result in greater conflict between these platform providers and 
faculty members. With many faculty members wanting to share their content for free with other 
universities, their goals will increasingly diverge from those of the platform providers, driving them to 
seek out other like-minded entities with whom to partner. 

Peercy at the University of Wisconsin-Madison is a strong proponent of globalized standards for 
engineering education and cross-border collaboration to deal with global problems of water, energy, 
pollution and resource distribution. With their international audiences, Peercy anticipates that MOOCs 
may allow for de facto standardization and could offer opportunities for multi-national efforts to 
address such global issues. 

Alumni Relations 

Several interviewees suggested that MOOCs can help alumni reconnect with their alma maters and with 
one another. A goal of the Social Media cMOOC created by the Department of Communication at the 
University of West Virginia was to provide continuing education for alumni. Sreenivasan, formerly at 
Columbia University, also mentioned "retraining opportunities for alumni" as a goal for Columbia's 
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MOOCs. Fisher at Vanderbilt University perceives an opportunity for MOOCs to strengthen connections 
with alumni: 

They are not paying just for four years but for a lifetime of learning possibilities that Vanderbilt 
can offer them. Maybe we charge other people for MOOCs but not our alumni, or we provide 
them other affordances like on-campus experiences as part of the MOOC. MOOCs could change 
this relationship and our definition of alumni: they would never stop being our students. (Fisher, 
Vanderbilt University). 

An evaluator at a large Midwestern university noted that using MOOCs to connect with alumni could 
help address sustainability of MOOC costs if ongoing participation in these courses encourages the 
alumni to donate more, either directly to the MOOC initiatives, or more generally to the institution. 

Community Outreach 

Several interviewees mentioned opportunities for MOOCs to be used in the future to strengthen 
connections with surrounding communities. For example, Fisher at Vanderbilt University noted that 
many of Vanderbilt's faculty members from the school of education already work with teachers in 
surrounding K-12 classrooms. Fisher envisages that these connections could be leveraged to implement 
a closed instance of a MOOC to improve the level of computer science education for the K-12 students. 
Vanderbilt University undergraduates could visit the classrooms on a weekly basis as "journey 
instructors" to supplement the online learning. 
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Conclusions and Recommendations 


It is widely felt that higher education is in the midst of a significant change (e.g., Bowen, 2013; Barber, 
Donnelly, & Rizvi, 2013) and must accommodate budget shrinkages and the changing needs of learners 
and employers. High expectations abound for the role MOOCs can play in this transformation. This 
paper identifies six major goals that institutions hope to achieve through MOOC initiatives. 

The first goal is to extend reach of the institution to a wider audience and improve access to education. 
Data from MOOC platforms indicate that this goal has been met to some extent in terms of geographical 
spread, but less so in terms of reaching individuals with fewer educational opportunities. Participants 
are mostly limited to highly motivated learners who have access to high bandwidth Internet connections 
and, perhaps not surprisingly, the majority has already earned at least a bachelor's or a master's degree, 
or even a doctorate. These may be new audiences for many existing college and university courses and 
programs, but it appears that MOOCs are mostly educating the educated and are therefore increasing 
the divide between those who have access to education and those who do not. It may be more accurate 
to state that they are extending access to life-long education rather than making education accessible to 
a broad group of people. On the other hand, this educated audience is also more likely to be able to pay 
for certificates and other services, allowing the production and delivery of MOOCs to be financially 
sustainable. While many institutions express the desire to improve access to education, and this is 
indubitably a laudable goal, there is little real incentive or financial justification for institutions to do so, 
except for those who can pay for it. 

Going forward, if institutions genuinely wish to broaden access to less educated audiences, they must 
identify multiple channels of communication to reach potential recruits. For example, they may need to 
use social media networks and advertise through high schools, employment agencies, or community 
organizations in the U.S. and abroad. Additionally, courses must be designed to serve less self-directed 
learners by incorporating motivational features and optional instructional scaffolding to address 
differences in participant preparation levels. However, as higher education institutions can ill afford to 
offer free and unsubsidized educational opportunities to the world, the costs of developing MOOCs will 
need to be covered by some fraction of participants who have adequate resources. 

The second goal is to build and maintain brand. If success in achieving this goal was measured purely by 
media publicity surrounding MOOCs, many institutions have indeed become more visible publicly as a 
result of their MOOC activities. Mostly this has been positive, but not always. Although several 
institutions suggested anecdotally that this goal has been at least partially accomplished, measuring 
success more tangibly could be achieved through comparing pre- and post-MOOC metrics on student 
recruitment and enrollment statistics, success in faculty recruitment and retention, breadth and 
generosity of donors, and quantity and size of grants. In addition, the costs of MOOC production and 
delivery should be compared to other strategies known to be effective at building and maintaining 
brand in order to ascertain which alternatives are most cost-effective. 

Typical of any early-stage, technology-based innovation, achievement of the third goal, to improve the 
economics of higher education, is still elusive for the most part. We found scant evidence that MOOCs 
have increased revenues substantially, and substantial evidence that they have increased costs 
significantly. Gradually materializing revenues could help shift the balance over the next few years but, 
unless MOOC producers and platform providers continue to expand ways to confer economic value on 
MOOC completion, for example, in the form of employer-recognized credentials, the market and 
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associated revenue streams will remain limited. One way or another, MOOCs will need to prove they 
can solve a problem or provide a service more efficiently than existing alternatives. 

We found that costs of developing MOOCs are very high and the process demands a great deal of 
personnel time and effort. Course design and delivery has shifted from a solo endeavor to team efforts 
including administrators in offices of digital technology, instructional designers, instructional 
technologists, videographers, and project managers. Faculty members themselves are, in most cases, 
being vastly underpaid for the opportunity costs of their time to develop MOOC content. While their 
primary intentions may be to improve opportunities for the educationally underserved, they are 
perhaps most effectively subsidizing the salaries of other employees at their own institutions or at the 
MOOC platform providers. Currently, there is little documentation to help ascertain whether costs of re- 
offering a MOOC diminish substantially as compared with costs of the initial offering. 

MOOCs could potentially reduce the costs of higher education if they can be used by academic 
institutions to eliminate the reproduction of similar courses across many campuses. If the licensing 
issues surrounding the sharing of materials across multiple campuses can be resolved, and if MOOCs can 
be re-run several times without significant adjustments at each offering, the high costs of initial MOOC 
development could be amortized over all the instances of use. But given that, in education, costs are 
mostly personnel-related, overall costs can only fall if personnel numbers or salaries fall first. Reduction 
of costs could be realized if academic institutions are willing to shift more of the responsibility for 
instructional support away from tenured faculty members and onto non-tenured instructors, teaching 
assistants, or outsourced personnel. Tenured faculty positions could not be easily eliminated in the short 
term, but they could be reduced over time, contributing to the current long-term trend towards 
"adjunctization" of higher education. This trend does not, however, bode well for the capacity of 
academic institutions to further various academic disciplines through research, nor is it likely to be 
welcomed by faculty members and their unions. For now, the picture being painted to keep the peace is 
one in which repetitive activities such as delivering the same lecture every year or grading uniform 
assignments are turned over to the MOOC, while faculty members spend time in the classroom with 
students on problem-solving, discussion, and debate. This approach may prove to be more educationally 
effective than current practice, but it will also be more expensive and therefore more financially 
unsustainable than an already unsustainable system. 

From the perspective of learners, MOOCs will only favorably impact the economics of obtaining an 
education if they remain free or charge minimal fees, but can be used to substitute costlier on-campus 
or online courses to obtain college credits or other credentials that will be recognized by employers. 
While this scenario may be beneficial to the students and to the taxpayer, it may not be in the interests 
of the academic institutions themselves as this will imply a loss of revenue per student, which may or 
may not be possible to make up by increasing enrollments. State and federal education policymakers 
could adjust regulations to create pathways for MOOCs to be accepted for credit in high schools or to 
satisfy government-mandated continuing education for professionals. In higher education it is less clear 
how accreditation agencies could be persuaded to move in this direction. Establishment of an 
accrediting organization for MOOCs and other non-traditional educational experiences would allow 
learners to accumulate a portfolio of credentials that serve as a viable supplement or alternative to a 
college degree. Furthermore, such accreditation of individual courses or other educational experiences 
should confer the ability to use public funds towards the costs of these credentials. Fain (2014) reports 
on two potential efforts in this direction and highlights some of the surrounding regulatory issues. 
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Ironically, MOOCs have introduced possibilities that, should they materialize, could significantly 
compete with the very institutions that are racing to develop them. 

Many interviewees would readily assert that the fourth goal of improving educational outcomes has 
been achieved through MOOCs, at least for a limited group of learners. Primarily these positive reports 
arise in situations where MOOCs have been integrated with on-campus courses or where on-campus 
courses have been re-designed to incorporate MOOC-like components. Documentation of a small 
number of these experiments has shown early signs of improving student performance as measured by 
assessments and course completion rates. Typical strategies include flipping the classroom, 
implementing frequent assessments, and spending class time on problem-solving activities, often in 
small groups, rather than on lecture. 

Little effort has been made to ascertain whether participants in stand-alone MOOCs gain useful skills 
and knowledge that can be applied in productive, real-world contexts. To determine whether the goal of 
improving educational outcomes can indeed be achieved through MOOCs alone, will require rigorous 
evaluation employing pre- and post-assessments of knowledge and skills, and comparisons with the 
outcomes of face-to-face or other online courses. Educators must get beyond the notion that course 
completion per se equates with learning and consider how to allow learners to demonstrate valuable 
competencies, both cognitive and non-cognitive, that may be acquired through MOOC participation. 
Longitudinal studies tracking post-MOOC outcomes such as sequences of courses taken, professional 
certifications obtained, or job opportunities received would help assess the longer term value of 
participating in these courses. More effort is also required to assess whether MOOCs as stand-alone 
experiences can be designed effectively to increase knowledge and cultivate skills for any but the most 
motivated learners. Several interviewees noted that there is much to be learned from the existing 
research on best practices in online learning, and that MOOC producers have for the most part failed to 
take advantage of accumulated expertise in this area. A useful exercise would involve reviewing with 
these experts of online learning what pedagogical practices can be usefully migrated from regular online 
courses to MOOCs. 

The goal of innovation in teaching and learning has certainly been met at institutions where online 
learning did not previously exist, as evidenced by the sheer number of new courses produced and 
course re-design efforts undertaken. Metrics on the frequency of these efforts could be compared with 
pre-MOOC activity to provide an assessment of how well the goal of innovation has been achieved, but 
it is not clear that innovation per se is a sufficient aspiration. In many cases, institutions are innovating 
with a further goal in mind, such as improving educational outcomes or increasing revenues. Assessing 
impact of MOOC initiatives on these distal goals would be a more worthwhile endeavor in judging their 
contribution to higher education. However, some institutions are unclear as to why they are embarking 
on MOOC initiatives and until they can agree internally on suitable and realistic goals, they will struggle 
to justify the expense and effort. For institutions that have been offering online and hybrid courses for 
many years, MOOCs represent more of an incremental step along a pre-existing trajectory than a major 
innovation. Some even consider MOOCs to be regressive because they fail to incorporate best practices 
in distance learning and repeat many of the mistakes of earlier attempts to educate at scale. 

Finally, the goal of conducting research on teaching and learning has met with many obstacles, but a few 
well-funded institutions with strong programming capacity have not been deterred from efforts to use 
the "big data" from the course delivery platforms to document online learner behavior and to 
experiment with variations in pedagogical strategies. Regrettably, MOOCs are often serving as more of a 
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playground populated by captive participants within which researchers tinker with the artifacts rather 
than as an object of research to determine whether MOOCs are an effective and cost-effective means of 
educating a broad range of learners compared with existing models of education. 

A potentially useful application of the big data from MOOCs is to provide learner analytics. These can be 
used to iteratively improve courses or to catalyze the advent of high quality adaptive learning and 
personalized educational experiences which meet the needs of a variety of learners. It is apparent, 
however, that the useful application of data mining techniques to data from MOOC platforms could be 
facilitated by standardization of data formats across the various platforms. It is also clear that the high 
demand on resources to develop sophisticated adaptive learning mechanisms will require the 
establishment of working partnerships between educators, instructional designers, and programmers. 
Federal approval of cross-institutional IRB agreements could facilitate projects in which expertise and 
resources can be pooled. Additionally, clarifications regarding applicability of FERPA to MOOCs would 
alleviate current confusion over how to handle participant data. 

Given the considerable investment that MOOCs represent, we strongly recommend that prior to 
embarking on MOOC-related initiatives, institutions carefully consider their goals and whether MOOCs 
present a realistic and financially justifiable means to achieve them. Administrators at academic 
institutions should work with representative faculty members and, where relevant, their unions to 
establish a strategy for engagement and a process for production or adoption of MOOCs. Data collection 
protocols should be devised up front to measure relevant indicators both prior to and post MOOC 
engagement in order to permit an objective assessment of whether stated goals are being met. By 
tracking personnel time commitments with respect to MOOC-related efforts and accounting for 
displacement of other productive activities, such as teaching or grant writing, costs of such initiatives 
can be estimated and compared with the evidence of goal achievement to judge whether MOOCs are a 
worthwhile investment of limited educational resources. 

Based on what we heard from our interviewees regarding MOOCs of the future, we expect that MOOCs 
will evolve to more closely resemble regular online courses in that they will be targeted at specific 
audiences, will offer some form of credential, and will combine elements of both synchronous and 
asynchronous participation. While participation at some basic level may be offered free, it is likely that 
revenue-generating services will proliferate including verified certificates, employer recommendation 
services, online tutoring, peer study group selection and facilitation, face-to-face components with a 
local instructor, and more complex assessments that require human grading or personalized interaction. 
We expect convergence between the extremes of cMOOC and xMOOC pedagogy to combine the 
strengths of networked learning and "creation of artifacts" with structured progression through a series 
of activities designed to improve specific skills and knowledge. 

Much of the hype surrounding MOOCs may be subsiding but it is clear that the infrastructure and effort 
that has been poured into such initiatives are not likely to evaporate overnight. Whether MOOCs as they 
currently stand persist into the future is certainly debatable, but there is no doubt that online and 
hybrid learning are here to stay and that MOOCs have ignited many valuable inter-disciplinary and cross- 
institutional discussions about how best to improve intellectual capital. There remains much work to be 
done to document how well MOOCs and related interventions serve their intended purposes and 
whether they do so cost-effectively. 
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Appendices 

Appendix I: Institutional Affiliations and Roles of Interviewees 


Table A.l Institutional Affiliations and Roles of Interviewees 



Number of 
institutions 
represented* 

Administrators/ 

executives 

Faculty 

members 

Administrators/ 
faculty members 

Researchers 

Other 

roles** 

Total 

interviewees 

Public universities 

16 

13 

4 

2 

1 


20 

Private universities 

14 

7 

11 

4 

3 

1 

26 

Community colleges 

9 

10 





10 

Platform providers 

5 

5 



1 


6 

Research organizations 

7 

1 



7 


8 

Other for-profit 
education companies 

4 

5 





5 

Other institutions*** 

7 

5 


1 

1 

1 

8 

Total 

62 

46 

15 

7 

13 

2 

83 


* Note that while only one person was interviewed at the majority of institutions, at a few institutions several individuals were interviewed, for example, to 
include one or two administrators, one or two faculty members, and one or two researchers. 

**Other roles: 1 educational technologist; 1 foundation program officer 

***Other institutions: 1 museum (2 interviewees), 1 K-12 school district; 1 educational technology advocacy group; 1 higher education association; 1 venture 
capital firm; 1 foundation; 1 independent consultant. 
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Appendix II: 

MOOCs: Expectations and Reality Interview Protocol 


Interviewer Name(s): 


Mode of communication: (phone, Skype, face-to-face meeting and location): 


Interviewee Name: 


Job title: 


Institution: 


Date: 


Start time: 


End time: 


1. What is your institution doing currently with respect to MOOCs? 

2. What are the primary goals of your institution in pursuing MOOCs? 

3. What is your role in this work and how did this role develop? 

4. How do you and your institution define a MOOC? 

5. Are there different types of MOOC? 

What are their characteristics? (e.g., structure, purpose, enrollment level, whether there is any fee associated, 
whether any kind of credential is offered). 

If a credential is offered, how does this differ from regular course credentials? 

Why did you choose specific types of MOOCs for your purposes? 

6. For specific MOOCs: 

What are the educational objectives of the MOOC(s) you are offering? 

What educational outcomes are being measured? 

What other data are you collecting pre-, post- and during the course? e.g., enrollment, demographics of participants, 
reasons for taking course, participation, test scores, completion, post-course applications (networking, pursuit of 
further study, employer acceptance of any credentials). 

How is data collected from MOOCs being used to improve pedagogy either online or on-campus? 

7. What amount of effort, personnel and other resources are required for the development and delivery of MOOCs? 

8. What are the most important cost drivers? 

9. How do the costs related to development and delivery of MOOCs compare with those for face-to-face or traditional online 
courses addressing the same topics? 

10. Where do you see you see things going with MOOCs over the next 5 years? (generally and at your institution). 

11. How do you see MOOCs helping your institution meet its goals over the next five years? 
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Appendix III: 

What Institutions are Doing in the MOOCspace 

This appendix provides a brief summary of the MOOC-related activity of each institution included in our 
study, provided the interviewee agreed to be identified. Names and titles of each interviewee are listed 
as written by the interviewees on their Informed Consent forms. Six interviewees requested total or 
partial anonymity. The information provided, titles, and roles of interviewees were current at the time 
of the interview (June 2013 - February 2014) unless otherwise mentioned. 

We list institutions alphabetically within the following categories: private universities, public 
universities, community colleges, platform providers, other for-profit education companies, research 
organizations, and other institutions/individuals. 

Private Universities 


Athabasca University, Canada 

Athabasca University administered the grants made available under the MOOC Research Initiative 
funded by the Bill & Melinda Gates Foundation. The grants were provided to examine the effectiveness 
of different MOOC models for a variety of audiences in different contexts. 

• George Siemens, Professor, Centre for Distance Education, Athabasca University, Canada. 
Siemens created the first cMOOC, Connectivism and Connected Knowledge, with Stephen 
Downes in 2008, which was offered at the University of Manitoba, Canada (see page 31 Origins 
of the cMOOC) and has re-run this twice since. He also offers a MOOC on learning analytics 
every Spring. In December 2013, Siemens moved to the University of Texas at Austin to become 
Director of the Learning Innovation and Networked Knowledge Research Lab. 

Columbia University, NY 

Columbia signed an agreement with Coursera in Fall 2012. Three MOOCs were offered in the first wave: 
Natural Language Processing, MOS Transistors, and Financial Engineering and Risk Management. Five 
more courses were launched or planned for Fall 2013 and Spring 2014 in diverse areas including 
economics, virology, history, and earth sciences. In March 2014, the university announced a partnership 
with edX. 

• Michael Cennamo, Educational Technologist, Columbia Center for New Media Teaching and 
Learning, Columbia University. 

Cennamo has worked on the design and development of four of Columbia's Coursera MOOCs 
and is currently working on three edX MOOCs. 

• Vincent Racaniello, Professor, Columbia University. 

Racaniello offered a series of two MOOCs, Virology 1: How Viruses Work and How Viruses Cause 
Disease, based on his on-campus course. (See Case 1) . 

• Sree Sreenivasan, Former Chief Digital Officer and Professor, Columbia University. 

Sreenivasan coordinated online initiatives on the Columbia University campus. The role included 
cataloging online initiatives; sharing what was and was not working; finding ways to encourage 
and help faculty, departments and schools learn more about online learning and social and 
digital media. In August 2013, Sreenivasan became Chief Digital Officer at the New York City 
Metropolitan Museum of Art. 
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Cornell University, NY 

Cornell joined the edX consortium in May 2013. From the 12-14 proposals submitted by faculty 
members through an internal Request for Proposals process, a committee chaired by the Provost 
selected four courses for the first round that would represent the Cornell brand without competing with 
on-campus courses. The second round of MOOCs will be launched in Fall 2014. 

• David BenDaniel, Berens Professor of Entrepreneurship, Johnson School, Cornell University. 
BenDaniel was invited to teach a MOOC on entrepreneurship during a sabbatical visit to MIT but 
declined because MOOCs cannot accommodate his Socratic style of teaching. 

• Ted Dodds, Chief Information Officer and Vice President for Information Technologies, Cornell 
University. 

Dodds is responsible for the entire IT@Cornell community. Fie is tasked with rebalancing 
allocation of IT resources to maximize the return on investment. 

East China Normal University, China 

The University is not currently offering MOOCs. 

• Minghua Li, Professor, School of Public Administration, East China Normal University. 

Li, in his 2011 book Research on Demand, Supply and Accreditation System of Higher Education 
for Migrant Workers in China, predicted a shift from degree markets to course markets. He is 
advocating for a complex platform that supports his vision for "MOOCs-lnside" courses, which 
are a form of blended learning. 

Harvard University, MA 

Harvard is one of the founding partners of edX. As of December 2013, HarvardX had offered 21 MOOCs 
on subjects including history, literature, medicine, statistics, and computer science. 

• Chris Dede, Wirth Professor in Learning Technology, Harvard University. 

Dede conducts research on online immersive simulations and is planning to conduct a massive, 
online experience on the subject of peer learning to model evidence-based online pedagogy at 
scale that includes effective use of peer learning. 

Massachusetts Institute of Technology (MIT), MA 

MIT is one of the founding members of edX and has offered 18 MITx MOOCs as of December 2013. 
XSeries certificates are available for sequences of MITx courses within a specific domain. 

• Professor Michael J. Cima, MIT. 

Prof. Cima offered the MOOC Introduction to Solid-state Chemistry and has integrated elements 
of the MOOC into his on-campus course. (See Case 6 ) 

• Dave Pritchard, Cecil and Ida Greene Professor of Physics, MIT. 

Prichard offered the MOOC Mechanics Online twice using the LON-CAPA platform and 
Mechanics Review once via edX. He conducts research related to MOOCs through the group 
Research in Learning and Assessing and Tutoring Effectively (RELATE) (See Case 9 ). 

• Professor Sanjay Sarma, Director of Digital Learning, MIT. 

Sarma works with faculty members to encourage them to offer their courses online and to 
support them through the design and development of the courses on the edX platform. 
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New York Institute of Technology (NYIT), NY 

NYU offered a series of webinar-like "mini-MOOCs" for the New York State Teacher Centers, providing 
technology-oriented professional development to K-12 teachers. Each section in the series consisted of 
a 30-minute synchronous session using Blackboard Collaborate and a three-week asynchronous 
component using Moodle. 

• Stan Silverman, Professor and Director, Technology Based Learning, New York Institute of 
Technology. 

Silverman created and facilitated the mini-MOOCs. 

Rensselaer Polytechnic Institute (RPI), NY 

RPI as an institution has not committed to creating MOOCs. RPI is interested in MOOCs as an 
experiment, for example in flipped classrooms, but is not convinced that they affect its mainstream 
business at present. Some instructors are, of their own accord, using MOOC content in flipped classroom 
activities. RPI is developing a MOOC-like bridging course in Calculus for incoming students. 

• John E. Kolb, VP of Information Technology and Services and CIO, RPI, NY. 

Kolb provides leadership for acquisition, use, and growth of campus-wide information 
resources, services, and technology to support education, research, and administrative 
activities. 

Stanford University, CA 

xMOOCs originated at the Stanford campus in 2011 with courses offered by Andrew Ng, Jennifer 
Widom, Sebastian Thrun and Peter Norvig (see page 34 Origins of the Modern MOOC (xMOOC). The 
MOOC platform providers Udacity, Coursera, and NovoEd were each founded by Stanford University 
professors. 

• Amy Collier, Director of Digital Learning Initiatives, Stanford University. 

Collier heads the pedagogy team that works with faculty members to design instruction for 
online and blended learning environments. 

• Emily Schneider, Student Director, Lytics Lab, Stanford University. 

Schneider helped establish the Lytics Lab to bring together researchers to collaborate on 
research related to online learning or MOOCs. Schneider was a lead organizer of the moocshop 
in 2013, one of the first MOOC research conferences. 

• Candace Thille, Assistant Professor, Director of Open Learning Initiative, Senior Research Fellow, 
Office of the Vice Provost for Online Learning, Stanford University. 

Thille directed the Open Learning Initiative (OLI) at Carnegie Mellon University before moving to 
Stanford in 2013. She is currently building the functionality of the OLI platform into the Open 
edX platform to support the OLI open courses. 

• Joseph Jay Williams, Research Fellow, Lytics Lab, Graduate School of Education, Office of the 
Vice Provost of Online Learning, Stanford University. 

Williams works in the Lytics Lab conducting research on MOOCs and on Khan Academy and 
supporting the research of other graduate students in the Lab. 
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Teachers College (TC), Columbia University, NY 

TC offered its first MOOC in Fall 2013, Big Data in Education, hosted on the Coursera platform. 

• Ryan Baker, Associate Professor, Teachers College, Columbia University, NY. 

Baker created and taught Big Data in Education. 

• Yuan "Elle" Wang, MOOC Researcher, Teachers College, Columbia University. 

Wang previously taught a MOOC via GLR, an online learning company based in China. She was 
the teaching assistant for Big Data in Education and is conducting research using data collected 
during the MOOC. 

Tsinghua University, China 

Tsinghua University signed a contract with edX to offer MOOCs in Summer 2013 and has offered two 
courses: Principles of Electric Circuits and History of Chinese Architecture. The content videos are 
presented in Mandarin with English subtitles. Five more MOOCs are planned for Spring 2014. 

• Dr. Fengliang Li, Associate Professor, Institute of Education, Tsinghua University, China. 

Li is interested in MOOC research. 

Tufts University, MA 

Tufts University has not offered a MOOC. 

• Lawrence S. Bacow, President Emeritus, Tufts University. 

Bacow is currently collaborating with Bill Bowen, former President of Princeton University and 
the Mellon Foundation; Mike McPherson, President of the Spencer Foundation; and Kevin 
Guthrie, President of ITHAKA, on a series of papers exploring issues of governance related to 
MOOCs, and more generally on how technology may influence colleges and universities in the 
future. 

Vanderbilt University, TN 

Vanderbilt University is both a producer of MOOCs and a consumer in that it integrates MOOCs created 
by other institutions into its own courses. The University partnered with Coursera in August 2012 and 
initially offered five MOOCs. As of August 2013, a total of 12 MOOCs had been planned or offered. 

• Doug Fisher, Director of the Vanderbilt Institute for Digital Learning, Vanderbilt University. 
Fisher leads the Institute's work on research, innovation, and support for digital learning. The 
Institute funds the exploration of digital learning by faculty members and fellows, and oversees 
the production of MOOCs. Fisher integrated a "closed instance" of the MOOC Artificial 
Intelligence, created at University of California, Berkeley and offered via edX, into his own 
course on the topic. 


Public Universities 


City University of New York (CUNY), NY 

A number of CUNY colleges have experimented with various MOOC derivatives. Macaulay Honors 
College joined as a node in a "distributed online collaborative course" (DOCC) on feminism and 
technology. JustPublics@365, a partnership between the Graduate Center of the City University of New 
York and the Ford Foundation offered a participatory open online course. 
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• Ann Kirschner, University Dean, Macaulay Honors College. 

Kirschner is an author and speaker on education policy and strategy, and digital media. 

• Joseph Ugoretz, Associate Dean of Teaching, Learning and Technology, Macaulay Honors College 
(CUNY). 

Ugoretz offered a mini-MOOC: Alternate Worlds, Imagining the Future of Education. The course 
was structured as a MOOC and was open but not massive. 

Georgia Institute of Technology (Georgia Tech), GA 

Georgia Tech's Center for 21 st Century Universities and Georgia Tech Professional Education (GTPE) are 
creating about 20 MOOCs hosted by Coursera, mostly funded by the Bill & Melinda Gates Foundation. In 
2013, Georgia Tech signed a 3-year contract with Udacity to offer the Online Master of Science in 
Computer Science program (OMSCS) which began enrolling students in January 2014. The initiative 
received a $2 million gift from AT&T to cover start-up costs. 

• David White, Assistant Dean for Academic Programs and Executive Director of the Online M.S. in 
Computer Science, College of Computing, Georgia Tech. 

White works with faculty, administration, and Udacity to implement the OMSCS program. 

Georgia State University, GA 

The University System has signed contracts with Coursera and Desire2Learn. However, it is not presently 
developing MOOCs as an institution. Individual professors have created their own MOOCs or used parts 
of MOOCs created by others. In January 2013, the Georgia State University Senate passed a resolution 
that students claiming competencies based on their MOOC experiences can apply to take a test of these 
competencies for potential granting of credit. 

• Timothy Renick, Vice Provost and Chief Enrollment Officer, Georgia State University. 

Renick directs Georgia State's admissions, enrollment, scholarship aid, and student success 
programs. 

North Carolina State University, NC 

Two centers at the university experimented with MOOCs. Distance Education Learning Technology 
Applications (DELTA) offered an open online course (OOC), Digital ASIC Design, with Engineering Online, 
and the Electrical and Computer Engineering Department. Separately, the Friday Institute has offered 
multiple MOOC-Eds (MOOCs for educators). 

• Dr. Glenn Kleiman, Executive Director, Friday Institute for Educational Innovation, North 
Carolina State University. The Friday Institute is focused on digital learning and professional 
development in the K-12 sector. The institute has offered several MOOC-Eds: Planning for 
Digital Transition; Coaching Digital Learning: Cultivating a Culture of Change; and World Class 
Teaching. Additional MOOC-Eds under development will focus on disciplinary literacy and 
several topics in teaching mathematics. Further information is available as www.mooc-ed.org . 

Pennsylvania State University, PA 

A MOOC strategy group was formed in 2012 and a first wave of five MOOCs was offered on the Coursera 
platform in Fall 2013 and Spring 2014: Introduction to Art; Maps and the Geospatial Revolution; Change, 
Innovation and Creativity; The Epidemics: The Dynamics of Infectious Diseases; and Energy, the 
Environment, and Our Future. 
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• Lawrence C. Ragan, Director, The Center for Online Innovation in Learning (COIL), Penn State 
World Campus. 

Ragan has contributed to the establishment and management of Penn State World campus. He 
directs COIL's work of research, scholarship, technology innovation, and leadership 
development programming. COIL spearheads the research related to the MOOCs created by 
Penn State University. 

San Jose State University (SJSU), CA 

SJSU, in partnership with Udacity, offered Remedial Algebra, Introduction to College Level Algebra, and 
Introduction to College Level Statistics in Spring 2013 and Introduction to Psychology and Introduction to 
Java Programming in Summer 2013. The University also offered Circuits Analysis in a flipped classroom 
using materials from the MITx MOOC Circuits and Electronics, which was hosted on the edX platform. 

• Catheryn Cheal, Ph.D., Associate Vice President and Senior Academic Technology Officer, San 
Jose State University, CA. 

Cheal was closely involved in faculty development and management of the SJSU partnerships 
with Udacity and edX. 

University of California (UC), Berkeley, CA 

UC Berkeley is one of the contributing members of edX. MOOCLab, a three-year research initiative 
within the Berkeley Resource Center for Online Education (BRCOE), funds and develops MOOCs as 
vehicles for pedagogical research in online education. 

• Armando Fox, Professor, Electrical Engineering and Computer Science Department, Faculty 
Advisor to the MOOCLab, UC Berkeley, CA. 

Fox launched the first software engineering MOOC on Coursera, Software as a Service, in 
January 2012. The course has been offered five times and was moved to the edX platform in Fall 
2012 . 

University of California, Irvine, CA 

UC Irvine was the first UC campus to become involved in open learning and one of the first universities 
invited to create MOOCs by Coursera. UC Irvine launched six MOOCs in Spring 2013 on various subjects 
including personal financial planning, math, physics, and public health. Four of the courses were offered 
again in Fall 2013, plus two new courses in economics. Some UC Irvine MOOCs offer the Signature Track 
option and two math MOOCs have earned ACE credit recommendations. In addition to the Coursera 
MOOCs, UC Irvine also offered the MOOC Society, Science, Survival: Lessons from AMC's "The Walking 
Dead," based on the television series "The Walking Dead," on the Instructure Canvas Network platform. 

• Melissa Loble, formerly Associate Dean for Distance Learning, University of California, Irvine 
Extension. 

Loble provided leadership in curriculum development, instructional design, and the selection 
and utilization of underlying platforms and technologies. Irvine Extension is the continuing 
education arm of the University that offers online as well as on-site courses. 

Loble recently moved to Canvas Network where she is Senior Director, responsible for strategic 
planning and day-to-day operations. 
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University of California, Santa Barbara, CA 

UC Santa Barbara has not offered a MOOC. 

• Christopher Newfield, Professor of Literature and American Studies, University of California, 
Santa Barbara. 

Newfield taught one of the three courses offered in tandem with the Coursera MOOC History 
and Future of (mostly) Higher Education. He received a Bill & Melinda Gates Foundation grant to 
conduct research on MOOCs offered at San Jose State University in partnership with Udacity. 

University of Colorado System, CO 

The University of Colorado's Boulder campus became a Coursera partner in May 2013, with a System 
contract following later that year. The Boulder campus offered four physics MOOCs that ran parallel to 
the on-campus courses; two graduate level engineering courses, Parallel Electronics and Linear 
Programming; and one undergraduate level course in the humanities, Comic Books and Graphic Novels. 
The Colorado Springs campus offered the course Intro to Game Programming using C#. The Downtown 
Denver campus (School of Business) offered the course Global Energy Management. Two more Coursera 
courses are planned for Summer and Fall 2014 release: Deciphering Secrets: Unlocking the Manuscripts 
of Medieval Spain will be taught from the Colorado Springs campus, and Global Health Responder from 
the Anschutz Medical Campus. In addition to the Coursera courses, the university is planning to convert 
a popular outreach course entitled Mini Med School into a MOOC to be offered on the Canvas learning 
management system. 

• Deborah Keyek-Franssen, (Ph.D.), Associate Vice President for Digital Education and 
Engagement, University of Colorado System. 

Keyek-Franssen helps campuses in the development and implementation of strategic plans for 
digital education including teaching and learning with technology, online courses and programs, 
and MOOCs. 

University System of Maryland (USM), MD 

USM is participating in a study led by Ithaka S+R involving 20 tests to integrate online components from 
Coursera, OLI at Carnegie Mellon, and various Pearson Products into their courses across a variety of 
disciplines. USM is considering the development of a "USM-OOC," a system-level open online course, 
produced and hosted by one USM campus and made available to all system campuses. 

• MJ Bishop, Director, Center for Innovation and Excellence in Learning and Teaching (CIELT), 
University System of Maryland. 

CIELT was established in 2013 to promote academic innovation. The center conducts research in 
best practices, disseminates results, and supports the USM campuses in implementing academic 
practices. 

University of Maryland University College (UMUC), MD 

Being an adult learning college, UMUC prefers to be a consumer of MOOCs rather than a producer. For 
students completing MOOCs that have been recommended for credit by ACE, UMUC will award transfer 
credits provided the students show adequate evidence of competency. 

• Karen Vignare, Associate Provost, Center for Innovation in Learning, University of Maryland 
University College (UMUC). 
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Vignare leads the Center for Innovation in Learning which was established in 2013 as an 
incubator for new approaches, practices, and learning models to understand and improve online 
learning. 

University of Oklahoma, OK 

University of Oklahoma has partnered with NextThought to provide courses that are open to the public 
for free as well as available for credit to matriculated students who receive additional student support 
and services. In Fall 2013, the first six courses were launched in subjects including Philosophy, 
International Studies, Meteorology, Chemistry, and Statistics. Fourteen additional courses were offered 
in Spring 2014. A total of 20 MOOCs have been developed and offered in one year. 

• Kyle Harper, Senior Vice Provost, University of Oklahoma. 

Harper assists the Provost on major new academic initiatives and acts as a liaison to the new 
working group on digital technology. 

• Nick Hathaway, Vice President for Administration and Finance, University of Oklahoma. 
Hathaway serves as the Chief of Staff for the President's office and partners with Harper to 
move the MOOC initiative forward. 

University of Texas System, TX 

University of Texas System is a member of the edX consortium and contributed $10 million towards 
building several dozen courses over several years. It has offered eight MOOCs so far in subjects including 
mathematics, music, medicine, and globalization. University of Texas at Arlington College of Nursing has 
created a MOOC as part of the MOOC2Degree initiative (see Case 4 ). In Fall 2014, University of Texas at 
Austin is planning to experiment with a "freemium" model whereby a free version of a course will be 
offered via edX along with a paid, gated course that includes more assessments, student support, and 
course credit. 

• Steven Mintz, Executive Director, Institute for Transformational Learning, University of Texas 
System. 

The Institute is charged with expanding access to educational programs that will improve 
learning and reduce costs, and promote a culture of educational innovation throughout the 
University of Texas System. 


University of Wisconsin-Madison, Wl 

The University offered two MOOCs via Coursera in Fall 2013, Video Games and Learning and Markets 
with Frictions. 

• Dr. Paul S. Peercy, Dean Emeritus, College of Engineering, University of Wisconsin-Madison. 
Peercy is a part of the Global Engineering Deans' Council and a proponent of technology in STEM 
education globally for which demand exceeds supply of quality instruction. Over a period of 
around 10 years, Peercy oversaw his department's shift to the use of flipped classrooms. 

West Virginia University (WVU), WV 

WVU signed a contract with Coursera in Summer 2013 and a committee was formed to invite and 
review faculty member proposals for courses to be launched in 2014. 
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• Matthew Martin, Professor of Communication Studies, West Virginia University. 

In February 2013, the Department of Communication Studies offered a four-week cMOOC on 
social media facilitated by four faculty members including Martin. The department is planning to 
offer another cMOOC on the topics of influence and persuasion. 


Community Colleges 

Academic Senate for California Community Colleges, CA 

This is a nonprofit organization created for the promotion and advancement of public community 
college education in California. 

• Beth Smith, President, Academic Senate for California Community Colleges. 

Smith represents 50,000 faculty members at community colleges in California. Smith expects 
that many community college faculty members will to try to adapt MOOC content to test how it 
works within their own classes but she cautions that there are a number of "unresolved issues" 
with respect to offering MOOCs as stand-alone online courses. Smith is also a Professor of 
Mathematics at Grossmont Community College, CA, which has not engaged in MOOC activity. 

Cuyahoga Community College (Tri-C), OH 

Tri-C offered a four-week developmental math MOOC, Believe Island, hosted on Blackboard CourseSites 
and funded by the Bill & Melinda Gates Foundation. The MOOC was offered four times in Summer 2013 
and remains open without facilitation (see Case 2) . 

• Belinda Miles, Provost and Executive Vice President of Academic and Student Affairs, Cuyahoga 
Community College, OH. 

Miles actively supported and led the Tri-C MOOC initiative. 

• A. Sasha Thackaberry, District Director, eLearning Technologies, Cuyahoga Community College, 
OH. 

Thackaberry was the project lead and created the first module of the developmental math 
MOOC as a prototype. She supports innovative eLearning initiatives across the college to 
provide solutions to institutional challenges. 

Foothill College, CA 

Foothill College is not producing or using a MOOC at present. 

• Judy Baker, Ph.D., Dean, Foothill College, CA. 

Baker has been active in the Open Education Resources (OER) movement since its inception and 
has offered workshops over the years to community college personnel on OER and online 
learning, including MOOCs. 

Inver Hills Community College, MN 

Inver Hills Community College is not currently producing or using MOOCs. 

• Christina Royal, Chief Academic Officer/Provost, Inver Hills Community College. 

In her previous role as Associate Vice President for eLearning and Innovation at Cuyahoga 
Community College (Tri-C), Royal participated actively in Tri-C's initiative to create a 
developmental math MOOC. Currently Royal is developing a strategic plan for online and adult 
learning at Inver Hills Community College. 
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Mt. San Jacinto Community College, CA 

In 2013, Mt. San Jacinto Community College offered a developmental English MOOC, Crafting an 
Effective Writer, on the Coursera platform. The project was funded by the Bill & Melinda Gates 
Foundation. 

• Patricia James, Dean of Distance Education, Mt. San Jacinto Community College, CA. 

James is an administrator for instructional technology, and distance education programs. She 
was actively involved in the design and development of Crafting an Effective Writer. 

Oklahoma City Community College, OK 

The college is evaluating MOOCs and other variations including SMOCs but is unwilling to commit scarce 
resources to an initiative with uncertain outcomes for its students. 

• Paul Sechrist, President, Oklahoma City Community College. 


Santa Barbara City College, CA 

The college is not producing or using a MOOC at present but provides many online courses for its 
students. 

• Dr. Douglas Hersh, Dean, Educational Programs, Santa Barbara City College, CA. 

Hersh conducts research on human presence design for online learning and has facilitated a 
move towards a unified learning management system for the college. He is a proponent of the 
establishment of a state-wide system to allow community college students to take online 
courses from any of the state's 112 community colleges. Since the time of our interview, Hersh 
moved to Grossmont College where he is Dean, Learning and Technology Resources. 

Wake Technical Community College, NC 

The college partnered with Udacity to create the MOOC Introductory Algebra Review, supported by the 
Bill & Melinda Gates Foundation, to help students prepare for the North Carolina community college 
placement test. The MOOC was opened in May 2013 and will remain open for 5 years. 

• Laura Kalbaugh, Dean, Wake Technical Community College, Raleigh, NC. 

Kalbaugh led the project to create Introductory Algebra Review. 


Platform Providers 


Coursera, CA 

Coursera is the largest for-profit MOOC platform provider, founded by computer science professors 
Andrew Ng and Daphne Koller from Stanford University. 

• Pang Wei Koh, Director of Course Operations, Coursera. 

Koh oversees the design, implementation, and support of all online courses and works with 
faculty and staff from partner institutions. 

edX, MA 

edX is a nonprofit entity founded by MIT and Harvard University and managed as a separate entity from 
the universities. The edX consortium includes 26 contributing member institutions. MOOCs constitute 
one part of a broad strategy for reinventing online and campus education. 
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• Rebecca Petersen, Director, Research and Educational Initiatives, edX. 

Petersen leads experimental initiatives and pilots the use of MOOCs in various academic 
settings, and promotes blended learning models among K-12 and higher education institutions. 

• Howard Lurie, formerly Vice President for External Affairs, edX; currently Managing Director, 
Education Partnerships, Acrobatiq - A Carnegie Mellon University Venture. 

FutureLearn, United Kingdom (UK) 

FutureLearn is a company wholly owned by The Open University in the UK, and provides platform 
services for MOOC production and delivery to 23 university partners. Additionally, three content 
partners - the British Library, British Museum, and British Council - work with some university partners 
to create courses that are offered in tandem with their exhibitions and other events. 

• Mark Lester, Head of UK Partnerships, FutureLearn. 

Lester, previously at The Open University, is responsible for recruiting partners and supporting 
them in the use of the FutureLearn platform. 

NextThought, OK 

NextThought has built a platform that encourages and facilitates connection among participants in an 
online learning community and can be used to deliver MOOCs. NextThought has helped develop and 
deliver 20 online courses for the University of Oklahoma. 

• Ken Parker, CEO, NextThought. 

Parker is the co-founder of NextThought. 

Straighterline, MD 

Straighterline does not offer MOOCs but offers online courses, all with ACE credit recommendations, on 
a subscription basis. The organization licenses materials from content providers including McGraw-Hill, 
ThinkWell, and Saylor Foundation, and provides supplementary services such as academic support, 
student advising, proctoring, pathways to credit, and transcripting. Articulation agreements with 
academic institutions guarantee acceptance of some courses for credit. Students can try the first two 
lessons of any Straighterline course for free. 

• Burck Smith, CEO, Straighterline. 

Smith is the founder of Straighterline and writes about education policy. 


Other For-profit Education Companies 


Academic Partnerships (AP), TX 

AP launched the MOOC2Degree initiative in January 2013 which converts the initial core course of a 
partner university's online degree program into a MOOC. Completers who are subsequently accepted 
into the partner university's full program can receive credit for the MOOC based upon criteria 
established by the university. AP helps to adapt the courses for delivery on any platform of the partner 
university's choice, and markets the courses to prospective students. 

• Dr. Justyna Dymerska, Executive Vice President, Academic Partnerships. 

Dymerska directs AP's international operations and strategy. 
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Kaplan Inc., USA 

Kaplan University is working on "large, open, zero-variable-cost-delivery pilots" and is planning to create 
a MOOC on the application of learning science at scale to instructional environments. It will be based on 
Kaplan's internal training program for instructional designers. 

• David Niemi, VP, Measurement and Research, Kaplan Inc. 

Niemi leads efforts to improve the quality of measurement, evaluate the effectiveness of 
curricula and instruction, and study the impact of innovative products and strategies. 

• Bror Saxberg, Chief Learning Officer, Kaplan Inc. 

Saxberg leads the Kaplan Innovation group that will design the MOOC. 

McGraw-Hill Education, NY 

The organization is not developing any platform that is targeted specifically to the delivery of MOOCs, 
but is working with institutions developing MOOCs and with MOOC platform providers to help them 
apply appropriate pedagogical strategies and to design learning interactions. 

• Stephen Laster, Chief Digital Officer, McGraw-Hill Education. 

Laster is responsible for the organization's e-learning and educational technology strategy. 

Pearson, NY 

Pearson is not directly creating MOOCs, but some Pearson business units provide infrastructure support 
to other institutions. For example, Embanet Compass helps many institutions provide their online 
courses. Pearson has also invested significantly in Coursera via its investment in Learn Capital. 

• Saad Rizvi, Senior Vice President, Efficacy, Pearson. 

Rizvi is a co-author of the publications Oceans of innovation: The Atlantic, the Pacific, Global 
Leadership and the Future of Education (2012), and An Avalanche is Coming: Fligher Education 
and the Revolution Ahead (2013). 

Research Organizations 

American Institutes for Research (AIR), DC 

AIR is a behavioral and social science research and evaluation organization. 

• Jessica Heppen, Managing Researcher, American Institutes for Research. 

Heppen conducts research on online learning, mostly in the K-12 sector, and has engaged in 
conversations with Cousera about possible research collaborations. 

California State University (CSU), CA 

• Dr. John Whitmer, Program Manager, Academic Technology & Analytics Office of the Chancellor, 
California State University. 

Whitmer was a co-author of the report on San Jose State University's SJSU+ Augmented Online 
Learning Environment [AOLE] Pilot Project and is conducting research on Mt. San Jacinto's 
MOOC Crafting an Effective Writer. 

Note: We list CSU here rather than under public universities above because we interviewed Whitmer as 
a researcher of MOOCs, not as a representative of CSU. We are aware, however, that CSU created a 
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MOOC Joint Task Force to examine current and potential legislation regarding online courses and to 
ensure effective faculty voice regarding the role of MOOCs in universities. 

Community College Research Center (CCRC), Teachers College, Columbia University, NY 

CCRC has published some of the best-known studies on online learning in higher education. The center 
is not actively conducting research on MOOCs but some of the community colleges it is working with are 
experimenting with integrating MOOC materials into their face-to-face or hybrid courses. 

• Shanna Jaggars, Assistant Director, Community College Research Center, Teachers College, 
Columbia University. 

Jaggars manages research studies focusing on community college student success and improving 
student motivation. 

Note: While we list CCRC here separately from Teachers College above, we do not count it as a separate 
institution for the purposes of our frequency analysis of institutional goals for pursuing MOOCs. 

Contact North, Ontario, Canada 

Contact North is Ontario's distance education and training network. 

• Dr. Tony Bates, Research Associate, Contact North, Ontario. 

Bates advises institutions of higher education on online learning strategy. He has published 
books on technology and e-learning, including analyses of the costs of online programs. He 
served as a participant instructor in the cMOOC CHANGE offered by Stephen Downes, focusing 
on management in higher education. Bates maintains a blog on online learning and distance 
education resources. 


ITHAKA/lthaka S+R, NY 

In partnership with the University System of Maryland (USM), Ithaka S+R is conducting a study involving 
20 experiments to integrate online components from Coursera, OLI at Carnegie Mellon, and various 
Pearson Products into USM courses across a variety of disciplines. The study includes side-by-side 
comparisons of traditional and hybrid versions of large introductory level courses, as well as case studies 
of smaller, mostly seminar-style courses. 

• Kevin Guthrie, President, ITHAKA. 

Guthrie is a founder of ITHAKA. 

• Rebecca Griffiths, Program Director for Online Learning, Ithaka S+R. 

Griffiths oversees the Bill & Melinda Gates Foundation-funded study being conducted in 
partnership with USM. 


National Research Council (NRC), Canada 

NRC is the Government of Canada's research and technology institution. 

• Stephen Downes, National Research Council. 

Downes offered the first cMOOC, Connectivism and Connected Knowledge, with George Siemens 
in 2008. (See page 33 Origins of the cMOOC). He has offered other cMOOCs since then including 
Personal Learning Environments, Networks, and Knowledge { PLENK ) with George Siemens; 
Critical Literacies with Rita Cobb; CHANGE, a 30-week-long MOOC with multiple facilitators, 
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collaborators, and visiting experts; and The Future of Higher Education in partnership with 
Desire2Learn. Downes maintains a widely-read, online newsletter about online learning: 

OLDaily . 

Pittsburgh Science of Learning Center (PSLC), PA 

PSLC, based at Carnegie Mellon University, is one of the National Science Foundation Science of Learning 
Centers. It "leverages cognitive theory and computational modeling to identify the instructional 
conditions that cause robust student learning" (see LearnLab at PSLC ). Learning researchers at PSLC 
conduct research on learning theories, for example, by embedding features into Open Learning Initiative 
(OLI) courses. PSLC also established DataShop, "a central repository to secure and store research data" 
and "a set of analysis and reporting tools" for the learning science community (see PSLC DataShop ). 

• Ken Koedinger, Professor of Human-Computer Interaction and Psychology, Carnegie Mellon 
University. 

Koedinger has two decades of experience in educational research in educational technology and 
intelligent tutoring systems. He is the Director of PSLC. 

SRI International, CA 

SRI is an independent, nonprofit research institute. 

• Barbara Means, Center for Technology in Learning, SRI International. 

Means will be conducting a review of projects funded by the Bill & Melinda Gates Foundation 
under the Post-Secondary Success Group's technology-based initiative. Some of these include 
MOOCs. Means participated in reviewing proposals for the MOOC Research Initiative. 

Other Institutions/Individuals 

American Council on Education (ACE), DC 

ACE provides credit recommendations for the military, government agencies, and corporate entities, 
evaluating occupational training and experiences to discover embedded competencies that can be 
aligned with credit-bearing degree programs. Several MOOCs have been reviewed by ACE and 
recommended for credit. 

• Cathy Sandeen, Vice President, Education Attainment and Innovation, American Council on 
Education. 

Sandeen oversees ACE's national agenda and directs the Center for Education Attainment and 
Innovation. 

American Museum of Natural History (AMNH), NY 

AMNH offered three 4-week courses aimed at educators via the Coursera Teacher Professional 
Development program: Genetics and Society: A Course for Educators; The Dynamic Earth: A Course for 
Educators; and Evolution: a Course for Educators. 

• Dr. Ro Kinzler, Senior Director, Science Education, American Museum of Natural History, NY. 

• Dr. Robert Steiner, Director, Online Teacher Education Programs, American Museum of Natural 
History, NY. 

Steiner and Kinzler were heavily involved with directing the overall MOOC initiative, course 
design, and creation of materials. Kinzler served as an instructor in one course. 
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Digital Promise, Washington, DC 

Digital Promise is an independent, nonprofit organization authorized by the U.S. Congress to incentivize 
innovation in education. 

• Karen Cator, President and CEO, Digital Promise. 

Cator previously worked as Director of the Office of Educational Technology at the U.S. 
Department of Education. She has been actively following the MOOC phenomenon due to her 
interest in lifelong learning and educational innovation using technology. 

Harrison Central School District, NY 

The school district is not producing or using MOOCs at present. 

• Lawrence Paska, Director of Social Studies, Harrison Central School District, Harrison, NY; former 
State Education Agency Technology Policy Coordinator. 

In his previous role as Technology Policy Coordinator for New York State, Paska developed 
policies and initiatives to foster and implement educational technology across learning 
environments statewide. 

Sir John Daniel, Canada 

• From 1990-2001, Sir John Daniel was Vice-Chancellor of The Open University in the United 
Kingdom, which has recently created FutureLearn, a for-profit company, to deliver MOOCs. He 
was President and CEO of the Commonwealth of Learning from 2004 to 2012 and headed 
UNESCO's Education Sector from 2001-2004. He is currently an advisor to Academic 
Partnerships in Texas. Daniel has written extensively on the topics of technology in education, 
including online learning and MOOCs. 

Technology Crossover Ventures (TCV), CA 

This venture capital firm invests in the online education space broadly, including online education 
providers and "enablers," curriculum-oriented products, schools, and software companies that support 
the education industry. For example, TCV was an early investor in Embanet, which helps universities 
take their programs online, and sold the firm to Pearson in 2012. TCV is not currently investing in 
MOOC-related companies as they do not fulfill TCV's investment criterion of a minimum of $30 million in 
revenues. 

• Jake Reynolds, General Partner, Technology Crossover Ventures. 

Reynolds is a board member of K-12 Inc., the largest for-profit provider of K-12 online education 
in the United States. Previously Reynolds served as a board member of Capella University. 

William and Flora Hewlett Foundation, CA 

The foundation has supported many innovative, technology-related projects in education. For example, 
the Open Educational Resource movement and Peer 2 Peer University (P2PU), a nonprofit, online, open 
learning community which allows users to develop and deliver their own courses and to participate in 
courses developed by others. 

• Victor Vuchic, Program Officer Education, William and Flora Hewlett Foundation. 

Vuchic has been a strong proponent and supporter of Open Educational Resources for many 
years. Before "MOOC" became a buzzword, he was working with groups providing massive open 
online courses through P2PU. In March 2014, Vuchic founded V3 Consulting. 
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Appendix IV: 

Costs of Online Video Production 


Video production costs for MOOCs and other online courses vary dramatically. At one extreme an 
instructor may use a desktop camera and video-recording software such as Camtasia or ScreenFlow to 
record him/herself. At the other extreme, an external professional production team may be hired to 
video-tape live lectures or recording sessions, and to incorporate custom-made animations and graphics 
into the edited product. The former strategy usually results in "low production value" video and the 
latter in "high production value" video. (See Box 12 for an explanation of these terms). In between these 
extremes, an in-house production team may work with course instructors to capture and edit the video. 
Decisions as to which strategy to adopt are usually made based on time and cost considerations, 
availability of a dedicated team to work within the required deadline for course launch, and capacity to 
meet desired production values. An internal production team at an institution that is simultaneously 
developing many online courses must juggle many priorities. Costs of engaging an internal production 
team can be hard to isolate for a specific project unless detailed time logs are maintained. 

The video production costs shown in Table A.2 are based on the work being executed by a video 
production company partnering with a large R1 university in the development and delivery of multiple 
online courses over a one-year period. The courses are primarily targeted at the university's enrolled 
students who pay fees and receive credit for successful course completion. However, like MOOCs, the 
courses are open to any member of the public to participate in for free and without receiving credit. The 
production company indicated that the university's decision-making process with respect to course 
production was primarily driven by considerations of quality of teaching and outcomes, consistent with 
the university's core values, as opposed to price sensitivity. The mandate was to create high production 
value videos. 

For each 16-week course completed to date, a total of 5-14 hours of finished video has been created. 
Each weekly unit of a course typically includes five 8-minute clips of video. Table A.2 provides estimated 
costs for a course that involves the production of 10 hours of finished video. The total cost is $43,344 
and the major cost drivers are camera operators and equipment, video-editing, the project manager, 
and the production manager. While these costs may be approximated to around $4,300 per hour of 
finished video, it is likely that an external team will discount rates as expected engagement time 
increases. Readers may substitute their own prices and estimated hours of video and labor to calculate 
their own local costs. If the video content can be re-used on multiple occasions, the costs can be spread 
across these instances. 
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Table A.2 Production Costs for an Online Course with 10 hours of Finished Video 


Ingredient 

Cost per unit 

Cost assumptions for one 
course, 10 hours finished 
video 

Estimated cost for one 
course, 10 hours of 
finished video 

Management 

Project manager 

$90,000/year 
incl. benefits 

Annual salary + benefits 
spread over multiple courses 

$5,000 

Production manager 

$90,000/year 
incl. benefits 

Annual salary + benefits 
spread over multiple courses 

$5,000 

Video capture 

Camera operator with 
equipment 

$800- 

$l,200/day 

2 people, 5 days each, 
$l,000/day 

$10,000 

Audio technician with 
equipment 

$500/day 

1 person, 5 days 

$2,500 

Make-up artist 

$350/day 

1 person, 5 days 

$1,750 

Set components 

Highly 

variable 

$250 per course 

$250 

Labor for set assembly 

Variable 

2 interns, 2 days at $12/hour 

$384 

Video facility/location rental 

$100/hour 

Rental for 5 days 

$4,000 

Tele-prompter 

$200/day 

5 days of use 

$1,000 

Video editing and production 

Video-editor 

$100/hour 

88 hours of editing 

$8,800 

Intern to obtain/verify 
copyright permissions 

$15/hour 

40 hours 

$600 

Graphic designer to create 
custom graphics/animation 

$100/hour 

16 hours 

$1,600 

Intern to locate graphics, 
images/animations 

$15/hour 

50 hours 

$750 

Graduate Assistants to 
review video for content 
errors 

$20/hour 

20 hours 

$400 

Video transcripts 

$130/hour 

10 hours finished video 

$1,300 

GRAND TOTAL 



$43,344 
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Assumptions : 

A day is 8 hours long. While "efficiency" of shooting varies depending on factors including instructor's preparedness and 
confidence, it is assumed that to obtain 1 hour of usable video requires 4 hours of shooting time. The rule of thumb adopted 
here for editing time is twice the shoot time plus 1 day extra. 

Note: Travel and accommodation for a "destination" shoot or for engagement of a team that is not locally based would be 
additional. For 8 hours travel time and overnight accommodation for a team of 5, this could amount to $4,350, assuming one 
production manager and two camera operators each at $l,000/day, one audio technician and one make-up artist, with hotel 
and per diem at $100 per person for one night. 

Box 12: 

High Production vs. Low Production Values for Video 

Low production value video: image is not sharp, color is dull, audio is tinny and unclear; background is 
static and unchanging. The lowest production values generally arise when an un-manned camera is 
placed in the back of a classroom to record a lecture. A step up involves the instructor speaking into a 
web-cam on his/her computer screen with the occasional PowerPoint slide included. 

High production value video: aims for substantial visual variety and maximal viewer engagement. Two or 
more cameras are used, switching back and forth to provide different perspectives. Cameras may be 
placed on sliders to allow movement. One camera is likely to have a wide-angled lens to capture the 
larger picture while another provides close-ups so that the instructor appears to be talking directly to 
the viewer. The edited video incorporates high quality graphics, animations, supplementary audio and 
images to provide visual cues that help explain or enhance the verbal message being delivered. 

Roles of production team members 

Project manager: co-ordinates across all courses, liaising with and scheduling instructors and production 
team, ensuring textual resources and images are verified for copyright. 

Production manager: co-ordinates the video production team and directs the video collection, editing, 
and post-production. 

Camera operators and eguipment: provide and operate camera equipment; arrange and control lighting. 
A wide-angled camera is used to capture a group, 1-2 other cameras are focused on individuals. If the 
shoot is complex with a dynamic set up requiring lights to be moved around, a separate lighting 
technician may be required. 

Audio-technician: sets up the audio equipment and microphones (portable, lapel, or boom); monitors 
the recording equipment while the shoot is under way; helps synchronize audio with the video material. 

Make-up artist: helps instructor get "camera-ready," including coaching on clothes to wear. 

Video-editors: review hundreds of gigabytes of digital content from multiple takes, eliminate repetitions 
and errors in elicitation, and distill the message to be delivered. Incorporate "eye-candy:" images, 
graphics, and animations to help explain the content. Submit "proof draft" to instructor or graduate 
assistants to check for content errors. 

Source: U.S. video production company. 
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Appendix V: Sample Pre- and Post-MOOC Participant Surveys 

Penn State University 

[This survey was provided courtesy of Penn State University]. 


NOTE: We are assuming that Coursera will gather basic demographic and user analytics information on the MOOC participants. 

Pre-course Survey 

1. Please indicate your previous online learning experience by selecting all that apply. 

a. I have taken other MOOCs 

b. I have taken other online learning courses for academic credit 

c. I have taken other online learning courses but not for academic credit 

d. None of the above 

2. I am taking the course out of general interest, curiosity, or enjoyment, 

a. not at all important b. moderately important c. important 

d. very important e. absolutely critical 

3. 1 intend to watch of the lectures for this course. 

a. none b. some c. most d. all 

4. 1 intend to complete of the assessments for this course. 

a. none b. some c. most d. all 

5. I intend to earn a Statement of Accomplishment (or Verified Certificate) for this course, 
a. strongly disagree b. somewhat disagree c. neutral 

d. somewhat agree e. strongly agree 

How important were the following considerations in your decision to enroll in this course? 

6. 1 am interested in taking a course from this particular institution. 

a. not at all important b. moderately important c. important 

d. very important e. absolutely critical 

7. I am interested in taking a course from this particular professor, 
a. not at all important b. moderately important c. important 

d. very important e. absolutely critical 

8. 1 am interested in earning a credential. 

a. not at all important b. moderately important c. important 

d. very important e. absolutely critical 

9. 1 am interested in connecting with other students interested in this topic, 

a. not at all important b. moderately important c. important 

d. very important e. absolutely critical 

10. 1 have friends taking this course. 

a. not at all important b. moderately important c. important 

d. very important e. absolutely critical 

11. The course relates to my current academic program. 

a. not at all important b. moderately important c. important 
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d. very important 


e. absolutely critical 


12. The course relates to my current job responsibilities or company's line of business, 

a. not at all important b. moderately important c. important 

d. very important e. absolutely critical 

13. The skills from this course may be useful for obtaining a new job. 

a. not at all important b. moderately important c. important 

d. very important e. absolutely critical 

14. Which of the following best describes your previous experience in the subject area of this course? 

a. I am mostly new to this subject. 

b. I have explored this subject on my own. 

c. I have completed some coursework or have some work experience in this field. 

d. I have a degree or significant work experience in this field. 

e. [+ custom questions from instructor] 

15. Due to the large enrollments in a MOOC, there is little to no interactions between the course instructor and participants. 
Does this model meet your expectations? 

a. YES, I understand there may be no or limited interactions with the course instructor 

b. NO, I expect to have at least some interactions with the course instructor 

c. I do not have expectations either way 


Post-course survey 

(Administered by instructor at the end of course) 


Please rate each of the following: 


Course content 


1. Factual accuracy 

poor 

fair 

good 

very good 

excellent 

2. Depth of content 

poor 

fair 

good 

very good 

excellent 

3. Real-world applicability 

poor 

fair 

good 

very good 

excellent 

Course materials 

4. Lecture videos 

poor 

fair 

good 

very good 

excellent 

5. In-video quizzes 

poor 

fair 

good 

very good 

excellent 

6. Assessments 

poor 

fair 

good 

very good 

excellent 

7. Discussion 

poor 

fair 

good 

very good 

excellent 

Instructor/teaching staff 

8. Clarity 

poor 

fair 

good 

very good 

excellent 

9. Knowledge 

poor 

fair 

good 

very good 

excellent 

10. Presentation 

poor 

fair 

good 

very good 

excellent 

11. Responsiveness 

poor 

fair 

good 

very good 

excellent 


Overall Experience 

1. This MOOC met my expectations. 

a. strongly disagree b. somewhat disagree c. neutral 

d. somewhat agree e. strongly agree 

2. The promotional materials for this MOOC were accurate 

a. Yes b. No 
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12. Overall course experience 


poor fair good very good excellent 

13. 1 feel like I achieved my personal goals for this course, 
a. strongly disagree b. somewhat disagree c. neutral 

d. somewhat agree e. strongly agree 

If you did not achieve your goals for this course, which of the following factors would have helped? 
14. More interesting content 


a. strongly disagree 

b. somewhat disagree 

c. neutral 

d. somewhat agree 

e. strongly agree 

f. N/A 

15. More practically relevant 

content 


a. strongly disagree 

b. somewhat disagree 

c. neutral 

d. somewhat agree 

e. strongly agree 

f. N/A 

16. Fewer technical issues 
a. strongly disagree 

b. somewhat disagree 

c. neutral 

d. somewhat agree 

e. strongly agree 

f. N/A 

17. Reduced scope and length of course 


a. strongly disagree 

b. somewhat disagree 

c. neutral 

d. somewhat agree 

e. strongly agree 

f. N/A 

18. Content spread over longer time period 


a. strongly disagree 

b. somewhat disagree 

c. neutral 

d. somewhat agree 

e. strongly agree 

f. N/A 

Please rate each of the following: 


19. Course workload 
a. too light 

b. somewhat light 

c. just right 

d. somewhat heavy 

e. too heavy 


20. Course difficulty 
a. too easy 

b. somewhat easy 

c. just right 

d. somewhat difficult 

e. too difficult 


21. Course pacing 
a. too slow 

b. somewhat slow 

c. just right 

d. somewhat fast 

e. too fast 


To what extent do you agree or disagree with each of the following statements? 

22. 1 found the material interesting 


a. strongly disagree 

b. somewhat disagree 

c. neutral 

d. somewhat agree 

e. strongly agree 

f. N/A 

23. The course ran smoothly, with few technical difficulties 

a. strongly disagree 

b. somewhat disagree 

c. neutral 

d. somewhat agree 

e. strongly agree 

f. N/A 
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24. Students who complete this course should be awarded college credit, 
a. strongly disagree b. somewhat disagree c. neutral 

d. somewhat agree e. strongly agree f. N/A 


25. How many hours per week did you spend on this course? 

a. 0-5 hours b. 5-10 hours c. 10-15 hours 

d. 15-20 hours e. 20+ hours 


How likely would you be to: 

26. Revisit the course materials in the future? 

a. very unlikely b. somewhat unlikely c. neutral 

d. somewhat likely e. very likely 

27. Take this course again? 

a. very unlikely b. somewhat unlikely c. neutral 

d. somewhat likely e. very likely 

28. Take another course by the same instructors? 

a. very unlikely b. somewhat unlikely c. neutral 

d. somewhat likely e. very likely 


29. Recommend this course to a friend? 

a. very unlikely b. somewhat unlikely c. neutral 

d. somewhat likely e. very likely 


In a previous online learning experience I valued . . . (Select all that apply) 


30. Communications with the other course participants 

a. strongly disagree b. somewhat disagree c. neutral 

d. somewhat agree e. strongly agree f. N/A 

31. Communications with the course instructor 

a. strongly disagree b. somewhat disagree c. neutral 

d. somewhat agree e. strongly agree f. N/A 

32. Access to the course content 

a. strongly disagree b. somewhat disagree c. neutral 

d. somewhat agree e. strongly agree f. N/A 

33. Opportunities for team work 

a. strongly disagree b. somewhat disagree c. neutral 

d. somewhat agree e. strongly agree f. N/A 

34. What did you like best about this course? [open text field] 


35. What did you like least about this course? [open text field] 

36. Other suggestions/comments [open text field] 
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American Museum of Natural History 
Pre-Course Survey 

[This survey was provided courtesy of American Museum of Natural History] 

1. What is your professional role? 

a. educator b. student c. retired d. other (please specify) 

2. If you are an educator, what is your role? (select all that apply) 

a. I am a preschool teacher. 

b. I am an elementary school teacher. 

c. I am a middle school teacher. 

d. I am a high school teacher. 

e. lama higher education (college/university) instructor. 

f. I teach in a public school. 

g. I teach in a private school. 

h. I am a school administrator. 

i. lama museum/informal educator. 

j. I am an after-school educator. 

k. I am a summer program educator. 

l. lama media specialist/librarian. 

m. lama staff development coordinator. 

n. Does not apply. 

3. If you are a student, what kind of school do you attend? (select all that apply) 

a. public b. private c. home school d. high school 

e. college f. graduate school d. continuing education f. does not apply. 

4. What subject/s do you teach? (select all that apply) 

a. chemistry b. biology c. Earth science 

d. physics e. general science f. environmental science 

g. Special Education h. English Language Learners (ELL) i. other (please specify) 

j. does not apply 


5. 


For how long have you been a teacher? 

a. pre-service 

b. 1-2 years 

c. 3-5 years 

d. 6-9 years 

e. 10-19 years 

f. 20+ years 

g. Does not apply. 


6. Gender: 

a. Male b. Female c. Other 


d. Decline to answer. 


7. With which ethnic group do you identify? 

a. American Indian or Alaskan Native 
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b. Asian 

c. Black or African American 

d. Hispanics of any race 

e. Native Hawaiian or Other Pacific Islander 

f. White 

g. Race and Ethnicity unknown 

h. Two or more races 

i. Other (Include text box to self-identify.) 

j. Decline to answer. 

8. In what country do you live? (Drop-down option) 

9. If you live in the United States, please enter your zip code. (Open-ended text box) 

10. What is the highest level of academic study you have completed? 

a. High School Diploma / U.S. GED 

b. Associate's Degree 

c. Bachelor's Degree 

d. Master's Degree 

e. Professional or Doctoral Degree (Ph.D., Ed.D., J.D., M.D.) 

f. Does not apply. 

11. On average, how many hours do you plan to spend on this course each week? 

a. Less than 1 hour 

b. 1-2 hours 

c. 3-5 hours 

d. 6-8 hours 

e. 8+ hours 

12. Do you intend to earn a Statement of Accomplishment or Verified Certificate for this course? 

a. yes b. no c. I don't know 

13. Please select the level of importance that the following factors had in your decision to enroll in this course. (Options: Not 
important, Somewhat important. Important, Very important, N/A) 

a. My district is providing professional development credit for this course. 

b. To deepen my current knowledge of the subject area. 

c. To learn skills that will help with my career. 

d. To gain access to educational materials. 

e. To connect with other teachers, become part of an online community, or expand network. 

14. How did you learn about this course? (Select all that apply) 

a. AMNH website 

b. Coursera website 

c. search engine 

d. district or school 

e. friend or colleague 

f. online social network 
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g. blog/news articles/press coverage 

h. other (please specify) 

15. What is your prior experience with this subject? (Select all that apply) 

a. previous coursework/degree 

b. I teach this subject. 

c. science enthusiast 

d. personal experience 

e. I work in a related field. 

f. No prior academic, personal, or professional experience in this subject area. 

16. What is your experience with the American Museum of Natural History? (select all that apply) 

a. I have used resources from the Museum's website 

b. I have participated in an AMNH online course. 

c. I have participated in face-to-face teacher professional development at AMNH. 

d. I have visited the Museum. 

e. I have no prior experience with the Museum. 


American Museum of Natural History 
Post-Course Survey 


1. On average, how many hours did you spend on this course each week? 

a. less than 1 hour 

b. 1-2 hours 

c. 3-5 hours 

d. 6-8 hours 

e. 8+ hours 

2. How appropriate was the amount of work in this course? 

a. too much work b. about right c. not enough work 

3. This course helped me: (Likert scale: Strongly Disagree, Disagree, No Opinion, Agree, Strongly Agree) 

a. learn important concepts 

b. learn to analyze and critically evaluate ideas, arguments, and points of view about the topics covered 

c. increase my capacity to teach science 

d. deepen my understanding of the connection between science and society 

e. gain a broader understanding of the process of scientific discovery 

4. Which of the following course materials contributed to your learning? (Likert scale: Strongly Disagree, Disagree, No 
Opinion, Agree, Strongly Agree) 

a. essays 

b. videos 

c. discussion forums 

d. quizzes 

e. assignment 
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5 . 


I plan to use or adapt the following course materials for my own teaching: 

a. essays b. videos c. assignment d. classroom resources 


6. Please rate how much you agree or disagree with each of the following statements: (Likert scale: Strongly Disagree, 
Disagree, No Opinion, Agree, Strongly Agree) 

a. Participating in discussions was helpful in understanding weekly content. 

b. Participating in discussions increased my engagement with the course. 

c. I felt comfortable sharing ideas and viewpoints in the online discussions. 

7. Did you receive professional development credit from your district for taking this course? 

a. yes b. no c. I don't know d. does not apply 

8. How accessible was the level of the content given your background in science? 

a. much too difficult b. too difficult c. about right 

d. too easy e. much too easy 

9. Technical issues were addressed by Coursera staff in a timely manner. 

a. yes b. no c. does not apply 

10. Would you recommend this course to a friend or colleague? 

a. yes b. no c. maybe 

11. Would you take another AMNH course offered through Coursera? 

a. yes b. no c. maybe 

12. Would you take an AMNH online science course that is eligible for graduate credit? (For more information, please visit our 
Seminars on Science site.) 

a. yes b. no c. maybe 

13. Would you be willing to speak to an AMNH representative about your course experience? 

a. yes b. no 

14. Are you interested in face-to-face professional development offered by AMNH? 

a. yes b. no c. maybe 

15. Would you like to be added to the AMNH Education Newsletter list? 

a. yes b. no c. maybe 

16. What is your overall rating of this course? 

a. excellent b. above average c. average d. below average e. poor 

17. Please tell us what you feel were the best parts of this course, (open-ended) 

18. Please tell us how the course might be improved, (open-ended) 
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Appendix VI: 

List of Interviewees 

(in alphabetical order by last name) 


• Lawrence S. Bacow, President Emeritus, Tufts University, MA 

• Judy Baker, Ph.D., Dean, Foothill College, CA 

• Ryan Baker, Associate Professor, Teachers College, Columbia University, NY 

• Dr. Tony Bates, President/CEO, Tony Bates Associates Ltd., Vancouver, BC, Canada 

• David BenDaniel, Berens Professor of Entrepreneurship, Johnson School, Cornell University 

• MJ Bishop, Director, Center for Innovation and Excellence in Learning and Teaching, University 
System of Maryland 

• Karen Cator, President and CEO, Digital Promise 

• Michael Cennamo, Educational Technologist, Columbia Center for New Media Teaching and 
Learning, Columbia University 

• Catheryn Cheal, Ph.D., Associate Vice President and Senior Academic Technology Officer, San 
Jose State University, CA 

• Professor Michael J. Cima, MIT 

• Amy Collier, Director of Digital Learning Initiatives, Stanford University 

• Sir John Daniel 

• Chris Dede, Wirth Professor in Learning Technology, Harvard University 

• Ted Dodds, Chief Information Officer and Vice President for Information Technologies, Cornell 
University 

• Stephen Downes, National Research Council 

• Dr. Justyna Dymerska, Executive Vice President, Academic Partnerships 

• Dr. Li Fengliang, Associate Professor, Institute of Education, Tsinghua University, China 

• Douglas H. Fisher, Director of the Vanderbilt Institute for Digital Learning, Vanderbilt University 

• Armando Fox, Professor, Electrical Engineering and Computer Science Department, Faculty 
Advisor to the MOOCLab, UC Berkeley, CA 

• Rebecca Griffiths, Program Director for Online Learning, Ithaka S+R 

• Kevin Guthrie, President, ITHAKA 

• Nick Hathaway, Vice President for Administration and Finance, University of Oklahoma 

• Kyle Harper, Senior Vice Provost, University of Oklahoma 

• Jessica Heppen, Managing Researcher, American Institutes for Research 

• Dr. Douglas Hersh, formerly Dean, Educational Programs, Santa Barbara City College, CA. 
Currently Dean, Learning and Technology Resources, Grossmont College, CA 

• Shanna Jaggars, Assistant Director, Community College Research Center, Teachers College, 
Columbia University 

• Patricia James, Dean of Distance Education, Mt. San Jacinto Community College, CA 

• Laura Kalbaugh, Dean, Academic Success and Transition Resources, Wake Technical Community 
College, Raleigh, NC 
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• Deborah Keyek-Franssen, (Ph.D.), Associate Vice President for Digital Education and 
Engagement, University of Colorado System 

• Dr. Ro Kinzler, Senior Director, Science Education, American Museum of Natural History, NY 

• Ann Kirschner, University Dean, Macaulay Honors College at the City University of New York 

• Dr. Glenn Kleiman, Executive Director, Friday Institute for Educational Innovation, North 
Carolina State University. 

• Ken Koedinger, Professor of Human-Computer Interaction and Psychology, Carnegie Mellon 
University 

• Pang Wei Koh, Director of Course Operations, Coursera 

• John E. Kolb, VP of Information Technology and Services and CIO, Rensselaer Polytechnic 
Institute 

• Stephen Laster, Chief Digital Officer, McGraw-Hill Education 

• Mark Lester, Head of UK Partnerships, FutureLearn 

• Minghua Li, Professor, School of Public Administration, East China Normal University 

• Melissa Loble, formerly Associate Dean for Distance Learning, University of California, Irvine 
Extension; currently Senior Director, Canvas Network 

• Howard Lurie, formerly Vice President for External Affairs, edX; currently Managing Director, 
Education Partnerships, Acrobatiq - A Carnegie Mellon University Venture 

• Matthew Martin, Professor of Communication Studies, West Virginia University 

• Barbara Means, Center for Technology in Learning, SRI International 

• Belinda Miles, Provost and Executive Vice President of Access, Learning, & Success, Cuyahoga 
Community College, OH 

• Steven Mintz, Executive Director, Institute for Transformational Learning, University of Texas 
System 

• Christopher Newfield, Professor of Literature and American Studies, University of California, 
Santa Barbara 

• David Niemi, VP, Measurement and Research, Kaplan Inc. 

• Ken Parker, CEO, NextThought 

• Lawrence Paska, Director of Social Studies, Harrison Central School District, Harrison, NY; former 
State Education Agency Technology Policy Coordinator 

• Dr. Paul S. Peercy, Dean Emeritus, College of Engineering, University of Wisconsin-Madison 

• Rebecca Petersen, Director, Research and Educational Initiatives, edX 

• Dave Pritchard, Cecil and Ida Greene Professor of Physics, MIT 

• Vincent Racaniello, Professor, Columbia University 

• Lawrence C. Ragan, Director, The Center for Online Innovation in Learning, Penn State World 
Campus 

• Timothy Renick, Vice Provost and Chief Enrollment Officer, Georgia State University 

• Jake Reynolds, General Partner, Technology Crossover Ventures 

• Saad Rizvi, Senior Vice President, Efficacy, Pearson 

• Christina Royal, Ph.D., Chief Academic Office r/Provost, Inver Hills Community College 

• Cathy Sandeen, Vice President, Education Attainment and Innovation, American Council on 
Education 

• Professor Sanjay Sarma, Director of Digital Learning, MIT 
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• Bror Saxberg, Chief Learning Officer, Kaplan Inc. 

• Emily Schneider, Student Director, Lytics Lab, Stanford University 

• Paul Sechrist, President, Oklahoma City Community College 

• George Siemens, formerly Professor, Centre for Distance Education, Athabasca University, 
Canada. Currently Director of the Learning Innovation and Networked Knowledge Research Lab, 
University of Texas at Austin. 

• Stan Silverman, Professor and Director, Technology Based Learning, New York Institute of 
Technology 

• Beth Smith, President, Academic Senate for California Community Colleges 

• Burck Smith, CEO and Founder, Straighterline 

• Sree Sreenivasan, formerly Chief Digital Officer and Professor, Columbia University. Currently 
Chief Digital Officer at the Metropolitan Museum of Art, NY 

• Dr. Robert Steiner, Director, Online Teacher Education Programs, American Museum of Natural 
History 

• Sasha Thackaberry, District Director, eLearning Technologies, Cuyahoga Community College, OH 

• Candace Thille, Assistant Professor, Director of Open Learning Initiative, Senior Research Fellow, 
Office of the Vice Provost for Online Learning, Stanford University 

• Joseph Ugoretz, Associate Dean of Teaching, Learning and Technology, Macaulay Honors College 
(CUNY) 

• Karen Vignare, Associate Provost, Center for Innovation in Learning, University of Maryland 
University College (UMUC) 

• Victor Vuchic, formerly Program Officer Education, William and Flora Hewlett Foundation. 
Currently founder of V3 Consulting. 

• Yuan "Elle" Wang, MOOC Researcher, Teachers College, Columbia University 

• David White, Assistant Dean for Academic Programs and Executive Director of the Online M.S. in 
Computer Science, College of Computing, Georgia Tech 

• Dr. John Whitmer, Program Manager, Academic Technology & Analytics Office of the Chancellor, 
California State University 

• Joseph Jay Williams, Research Fellow, Lytics Lab, Graduate School of Education, Office of the 
Vice Provost of Online Learning, Stanford University 

• Anonymous 

• Development Officer, Highly Selective University, U.S. 

• Online Learning Coordinator, Large California Community College 

• Professor, East Coast University, U.S. 

• Senior Administrator, Leading UK University 

• Senior Administrator, East Coast University, U.S. 
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